GitHub user tcrayford opened a pull request:
https://github.com/apache/kafka/pull/3398
allow transactions in producer perf script
allow the transactional producer to be enabled in `producer-perf.sh`, with
a new flag `--use-transactions`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/heroku/kafka
allow_transactions_in_producer_perf
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/3398.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3398
----
commit 9170e87b2a154e7478af0ede43cc56b01f75bedd
Author: Xavier Léauté <[email protected]>
Date: 2017-05-19T00:02:51Z
KAFKA-5192: add WindowStore range scan (KIP-155)
Implements range scan for keys in windowed and session stores
Modifies caching session and windowed stores to use segmented cache keys.
Cache keys are internally prefixed with their segment id to ensure key
ordering in the cache matches the ordering in the underlying store for keys
spread across multiple segments.
This should also result in fewer cache keys getting scanned for queries
spanning only some segments.
Author: Xavier Léauté <[email protected]>
Reviewers: Damian Guy, Guozhang Wang
Closes #3027 from xvrl/windowstore-range-scan
(cherry picked from commit e28752357705568219315375c666f8e500db9c12)
Signed-off-by: Guozhang Wang <[email protected]>
commit b31df613f2664a8dc699b421b392f0aca9d5f83f
Author: Dana Powers <[email protected]>
Date: 2017-05-19T13:03:22Z
KAFKA-3878; Support exponential backoff policy via reconnect.backoff.max
(KIP-144)
Summary:
- add `reconnect.backoff.max.ms` common client configuration parameter
- if `reconnect.backoff.max.ms` > `reconnect.backoff.ms`, apply an
exponential backoff policy
- apply +/- 20% random jitter to smooth cluster reconnects
Author: Dana Powers <[email protected]>
Reviewers: Ewen Cheslack-Postava <[email protected]>, Roger Hoover
<[email protected]>, Ismael Juma <[email protected]>
Closes #1523 from dpkp/exp_backoff
(cherry picked from commit abe699176babe4f065b67b2b72d20daa0a2e46a1)
Signed-off-by: Ismael Juma <[email protected]>
commit 64e12d3c641baeb6a4a624b971ba33721d656054
Author: Ewen Cheslack-Postava <[email protected]>
Date: 2017-05-19T18:26:59Z
KAFKA-4714; TimestampConverter transformation (KIP-66)
Author: Ewen Cheslack-Postava <[email protected]>
Reviewers: Konstantine Karantasis <[email protected]>, Jason
Gustafson <[email protected]>
Closes #3065 from ewencp/kafka-3209-timestamp-converter
(cherry picked from commit 61bab2d875ab5e03d0df4f62217346549a4c64c3)
Signed-off-by: Jason Gustafson <[email protected]>
commit 037f63882df854c923bb9de11856318b951ecb32
Author: Matthias J. Sax <[email protected]>
Date: 2017-05-19T23:29:25Z
MINOR: improve descriptions of Streams reset tool options
Author: Matthias J. Sax <[email protected]>
Reviewers: Bill Bejeck, Guozhang Wang
Closes #3107 from mjsax/minor-reset-tool-options
(cherry picked from commit 338857e1c2c2bb0c6011fcab477818102b95cb56)
Signed-off-by: Guozhang Wang <[email protected]>
commit 55330cc2931cc05cb1172cc207ada60154a6250a
Author: Apurva Mehta <[email protected]>
Date: 2017-05-20T01:51:37Z
KAFKA-5269; Retry on unknown topic/partition error in transactional requests
We should retry AddPartitionsToTxnRequest and TxnOffsetCommitRequest when
receiving an UNKNOWN_TOPIC_OR_PARTITION error.
As described in the JIRA: It turns out that the
`UNKNOWN_TOPIC_OR_PARTITION` is returned from the request handler in KafkaAPis
for the AddPartitionsToTxn and the TxnOffsetCommitRequest when the broker's
metadata doesn't contain one or more partitions in the request. This can happen
for instance when the broker is bounced and has not received the cluster
metadata yet.
We should retry in these cases, as this is the model followed by the
consumer when committing offsets, and by the producer with a ProduceRequest.
Author: Apurva Mehta <[email protected]>
Reviewers: Guozhang Wang <[email protected]>, Jason Gustafson
<[email protected]>
Closes #3094 from
apurvam/KAFKA-5269-handle-unknown-topic-partition-in-transaction-manager
commit a144c5277dd95d8862ea6d9a497e5a087aa491fe
Author: Ewen Cheslack-Postava <[email protected]>
Date: 2017-05-22T01:04:32Z
HOTFIX: In Connect test with auto topic creation disabled, ensure
precreated topic is always used
Author: Ewen Cheslack-Postava <[email protected]>
Reviewers: Jason Gustafson <[email protected]>
Closes #3112 from ewencp/hotfix-precreate-topic
(cherry picked from commit d190d89dbc3df90e00f5e3c55507f67c5818504e)
Signed-off-by: Ewen Cheslack-Postava <[email protected]>
commit c0e789bdcc039a177f99bfecf36191f5be566d63
Author: Magnus Edenhill <[email protected]>
Date: 2017-05-22T01:23:12Z
MINOR: Fix race condition in TestVerifiableProducer sanity test
## Fixes race condition in TestVerifiableProducer sanity test:
The test starts a producer, waits for at least 5 acks, and then
logs in to the worker to grep for the producer process to figure
out what version it is running.
The problem was that the producer was set up to produce 1000 messages
at a rate of 1000 msgs/s and then exit. This means it will have a
typical runtime slightly above 1 second.
Logging in to the vagrant instance might take longer than that thus
resulting in the process grep to fail, failing the test.
This commit doesn't really fix the issue - a proper fix would be to tell
the producer to stick around until explicitly killed - but it increases
the chances of the test passing, at the expense of a slightly longer
runtime.
## Improves error reporting when is_version() fails
Author: Magnus Edenhill <[email protected]>
Reviewers: Apurva Mehta <[email protected]>, Ewen Cheslack-Postava
<[email protected]>
Closes #2765 from edenhill/trunk
(cherry picked from commit dc10b0ea014974b450f2abcd4a58e09d5b988e9e)
Signed-off-by: Ewen Cheslack-Postava <[email protected]>
commit 6d04e412cffabb1976aa935b4fb4767fd255836e
Author: Matthias J. Sax <[email protected]>
Date: 2017-05-22T05:16:18Z
KAFKA-4923: Modify compatibility test for Exaclty-Once Semantics in Streams
- add broker compatibility system tests
Author: Matthias J. Sax <[email protected]>
Reviewers: Damian Guy, Eno Thereska, Guozhang Wang
Closes #2974 from
mjsax/kafka-4923-add-eos-to-streams-add-broker-check-and-system-test
(cherry picked from commit 495836a4a27c2219f72a5f3776ba8e5216493b41)
Signed-off-by: Guozhang Wang <[email protected]>
commit ef88ee5045ea791d3208c3dcbc8723c4b2092262
Author: Mickael Maison <[email protected]>
Date: 2017-05-22T10:00:30Z
KAFKA-5294: PlainSaslServerFactory should allow a null Map
If props is null, use POLICY_NOPLAINTEXT default value: false
Author: Mickael Maison <[email protected]>
Reviewers: Ismael Juma <[email protected]>, Rajini Sivaram
<[email protected]>
Closes #3105 from mimaison/KAFKA-5294
(cherry picked from commit d3f1407f7bc95b823516e5a1bdaa23c717b6354e)
Signed-off-by: Rajini Sivaram <[email protected]>
commit e08e76b7166f5690cafdc8d5e52cd81746408a37
Author: Ismael Juma <[email protected]>
Date: 2017-05-22T12:37:29Z
KAFKA-5289: handleStopReplica should not send a second response
`shutdownIdleFetcherThreads()` can throw InterruptedException
for example.
Author: Ismael Juma <[email protected]>
Reviewers: Rajini Sivaram <[email protected]>
Closes #3096 from
ijuma/kafka-5289-stop-replica-should-not-send-two-responses
(cherry picked from commit ceb10c53302bd4824737e03ab86ca19e2aa64c7a)
Signed-off-by: Rajini Sivaram <[email protected]>
commit 4c29b1544bb058a004c57c3e559c7458810df2b4
Author: Ismael Juma <[email protected]>
Date: 2017-05-22T15:08:06Z
KAFKA-5303, KAFKA-5305: Improve logging when fetches fail in
ReplicaFetcherThread
Author: Ismael Juma <[email protected]>
Reviewers: Rajini Sivaram <[email protected]>
Closes #3115 from ijuma/kafka-5305-missing-log-info-replica-fetcher
(cherry picked from commit c2ced5fb51f2a4ec94d158e18836afb7284d26ce)
Signed-off-by: Rajini Sivaram <[email protected]>
commit 67f691bbd93b06efe58f2ba28f1b85efedf99651
Author: Vahid Hashemian <[email protected]>
Date: 2017-05-22T17:59:38Z
KAFKA-5277; Sticky Assignor should not cache previous assignment (KIP-54
follow-up)
... plus some minor cleanup
Author: Vahid Hashemian <[email protected]>
Reviewers: Jason Gustafson <[email protected]>
Closes #3092 from vahidhashemian/KAFKA-5277
(cherry picked from commit 65861a712ddf67eb071a00218730926fdeef7084)
Signed-off-by: Jason Gustafson <[email protected]>
commit a9c9dbf58f0e9ccf873e100b62f0f811ad5e4659
Author: Jason Gustafson <[email protected]>
Date: 2017-05-22T22:41:26Z
KAFKA-5186; Avoid expensive log scan to build producer state when upgrading
Author: Jason Gustafson <[email protected]>
Reviewers: Jun Rao <[email protected]>
Closes #3113 from hachikuji/KAFKA-5186
(cherry picked from commit fcdbb71953fc4c92559a9c7adb4cb8ad4a74acd6)
Signed-off-by: Jason Gustafson <[email protected]>
commit 9ae4deaa5724220569b0af312500130079b7fdb0
Author: Ewen Cheslack-Postava <[email protected]>
Date: 2017-05-23T00:46:23Z
HOTFIX: Replace JDK download and fix missing argument in Vagrant
provisioning script
Author: Ewen Cheslack-Postava <[email protected]>
Reviewers: Jason Gustafson <[email protected]>, Ismael Juma
<[email protected]>
Closes #3121 from ewencp/hotfix-vagrant-provisioning
(cherry picked from commit cca9ad42434265e388ec0100e0080250120e2cf9)
Signed-off-by: Ewen Cheslack-Postava <[email protected]>
commit 33216846fe070dfb5dd09eab9f53490bae92c357
Author: Jason Gustafson <[email protected]>
Date: 2017-05-23T03:00:07Z
MINOR: Broker should disallow downconversion of transactional/idempotent
records
Author: Jason Gustafson <[email protected]>
Reviewers: Guozhang Wang <[email protected]>
Closes #3118 from hachikuji/disallow-transactional-idempotent-downconversion
(cherry picked from commit e3e2f1d22d17a20ccccf67218c600e7e1647e1ca)
Signed-off-by: Jason Gustafson <[email protected]>
commit 2b326faf5f5a2bc632b842c26d2f48a07fa8a15e
Author: Apurva Mehta <[email protected]>
Date: 2017-05-23T16:45:31Z
KAFKA-5247; Materialize committed offsets in offset order
With this patch, offset commits are always materialized according to the
order of the commit records in the offsets topic.
Before this patch, transactional offset commits were materialized in
transaction order. However, the log cleaner will always preserve the record
with the greatest offset. This meant that if there was a mix of offset commits
from a consumer and a transactional producer, then it we would switch from
transactional order to offset order after cleaning, resulting in an
inconsistent state.
Author: Apurva Mehta <[email protected]>
Reviewers: Guozhang Wang <[email protected]>, Jason Gustafson
<[email protected]>
Closes #3108 from
apurvam/KAFKA-5247-materialize-committed-offsets-in-offset-order
(cherry picked from commit d1853f791147d66303c2700a8e58af762518188c)
Signed-off-by: Jason Gustafson <[email protected]>
commit 9e0581df513f2953bb7a899f784410bc99407465
Author: Damian Guy <[email protected]>
Date: 2017-05-23T16:50:45Z
MINOR: remove TransactionCoordinatorIntegrationTest
`TransactionCoordinatorIntegrationTest` is not covering anything that isn't
already covered by the more complete `TransactionsTest`
Author: Damian Guy <[email protected]>
Reviewers: Guozhang Wang <[email protected]>
Closes #3128 from dguy/minor-remove-test
(cherry picked from commit 5a6676bfca9f56c467ff46aefffc6f344ff4bf65)
Signed-off-by: Guozhang Wang <[email protected]>
commit fdbf2e5f7ad99b25feff2122cbcf4c51697250e3
Author: Jason Gustafson <[email protected]>
Date: 2017-05-23T16:53:18Z
MINOR: Log transaction metadata state transitions plus a few cleanups
Author: Jason Gustafson <[email protected]>
Reviewers: Guozhang Wang <[email protected]>
Closes #3081 from hachikuji/minor-add-txn-transition-logging
(cherry picked from commit 70ec4b1d927cad7373eda2c54ef44bfc4275832f)
Signed-off-by: Jason Gustafson <[email protected]>
commit 49a2482c22ca75516038ce961b5d053e9d2def56
Author: Matthias J. Sax <[email protected]>
Date: 2017-05-23T18:53:13Z
MINOR: improve EmbeddedKafkaCluster test utility for deleting topics
Author: Matthias J. Sax <[email protected]>
Reviewers: Bill Bejeck, Damian Guy, Eno Thereska, Guozhang Wang
Closes #3104 from mjsax/minor-improve-embedded-kafka-cluster
(cherry picked from commit 49587750570a2aaf332eb73f5aad373196f984ce)
Signed-off-by: Guozhang Wang <[email protected]>
commit e6600a36816718c90f896ab89bda6650de60ad78
Author: Guozhang Wang <[email protected]>
Date: 2017-05-23T20:34:43Z
KAFKA-5280: Protect txn metadata map with read-write lock
Two major changes plus one minor change:
0. change stateLock to a read-write lock.
1. Put the check of "isCoordinator" and "coordinatorLoading" together with
the return of the metadata, under one read lock block, since otherwise we can
get incorrect behavior if there is a change in the metadata cache after the
check but before the accessing of the metadata.
2. Grab the read lock right before trying to append to local txn log, and
until the local append returns; this is to avoid the scenario that the epoch
has actually changed when we are appending to local log (e.g. emigration
followed by immigration).
3. only watch on txnId instead of txnId and txnPartitionId in the txn
marker purgatory, and disable reaper thread, as we can now safely clear all the
delayed operations by traversing the marker queues.
Author: Guozhang Wang <[email protected]>
Reviewers: Jason Gustafson, Jun Rao
Closes #3082 from guozhangwang/K5231-read-write-lock
(cherry picked from commit 1bf64833168e81015e1f30a30c64eb6849c15422)
Signed-off-by: Guozhang Wang <[email protected]>
commit 69970630ba91de4a48b83a9101c487cd4eff3b46
Author: Onur Karaman <[email protected]>
Date: 2017-05-23T23:29:55Z
KAFKA-5310; reset ControllerContext during resignation
This ticket is all about ControllerContext initialization and teardown. The
key points are:
1. we should teardown ControllerContext during resignation instead of
waiting on election to fix it up. A heapdump shows that the former controller
keeps pretty much all of its ControllerContext state laying around.
2. we don't properly teardown/reset
ControllerContext.partitionsBeingReassigned. This can cause problems when the
former controller becomes re-elected as controller at a later point in time.
Suppose a partition assignment is initially R0. Now suppose a reassignment
R1 gets stuck during controller C0 and an admin tries to "undo" R1 (by deleting
/admin/partitions_reassigned, deleting /controller, and submitting another
reassignment specifying R0). The new controller C1 may succeed with R0. If the
controller moves back to C0, it will then reattempt R1 even though that
partition reassignment has been cleared from zookeeper prior to shifting the
controller back to C0. This results in the actual partition reassignment in
zookeeper being unexpectedly changed back to R1.
Author: Onur Karaman <[email protected]>
Reviewers: Jun Rao <[email protected]>
Closes #3122 from onurkaraman/KAFKA-5310
(cherry picked from commit beeddc25d6b443ed344658a2562a162fb03048ef)
Signed-off-by: Jun Rao <[email protected]>
commit 999c247e076cbf7e84141533c5a9042a844d57ae
Author: Ismael Juma <[email protected]>
Date: 2017-05-23T23:36:52Z
KAFKA-5135; Controller Health Metrics (KIP-143)
Author: Ismael Juma <[email protected]>
Reviewers: Jun Rao <[email protected]>, Onur Karaman <[email protected]>
Closes #2983 from ijuma/kafka-5135-controller-health-metrics-kip-143
(cherry picked from commit 516d8457d8142111a91af94cab918c84990685da)
Signed-off-by: Ismael Juma <[email protected]>
commit 44294a8941145fec2d973fd448ccbf97c86f0dd4
Author: dejan2609 <[email protected]>
Date: 2017-05-24T05:07:17Z
KAFKA-5081: Force jackson-annotations to a single version matching expected
Jackson version
**JIRA ticket:** [KAFKA-5081 two versions of jackson-annotations-xxx.jar in
distribution tgz](https://issues.apache.org/jira/browse/KAFKA-5081)
**Solutions:**
1. accept this merge request **_OR_**
2. upgrade jackson libraries to version **_2.9.x_** (currently available as
a pre-release only)
**Related jackson issue:** [Add explicit \`jackson-annotations\` dependency
version for
\`jackson-databind\`](https://github.com/FasterXML/jackson-databind/issues/1545)
**Note:** previous (equivalent) merge request #2900 ended up deep in the
sand with swarm of messages due to flaky test, so I opted to close it and to
open this one.
ijuma: FYI
Author: dejan2609 <[email protected]>
Reviewers: Ewen Cheslack-Postava <[email protected]>
Closes #3116 from dejan2609/KAFKA-5081
(cherry picked from commit 90dc2d16fbed59fc65e6e190380c8b9c43b6d95a)
Signed-off-by: Ewen Cheslack-Postava <[email protected]>
commit 23de4f4cdda23181fe0b925dd9feb2444dc06e3a
Author: Apurva Mehta <[email protected]>
Date: 2017-05-24T06:08:57Z
KAFKA-5273: Make KafkaConsumer.committed query the server for all partitions
Before this patch the consumer would return the cached offsets for
partitions in its current assignment. This worked when all the offset commits
went through the consumer.
With KIP-98, offsets can be committed transactionally through the producer.
This means that relying on cached positions in the consumer returns incorrect
information: since commits go through the producer, the cache is never updated.
Hence we need to update the `KafkaConsumer.committed` method to always
lookup the server for the last committed offset to ensure it gets the correct
information every time.
Author: Apurva Mehta <[email protected]>
Reviewers: Jason Gustafson, Guozhang Wang
Closes #3119 from
apurvam/KAFKA-5273-kafkaconsumer-committed-should-always-hit-server
(cherry picked from commit 4d89db968257c0c3273ece7e5546cb95ddcbeb46)
Signed-off-by: Guozhang Wang <[email protected]>
commit 308c7e47c3b8c4fa2835534b9d28dd9c2e2dc1a3
Author: Vahid Hashemian <[email protected]>
Date: 2017-05-24T16:14:10Z
MINOR: Improve the help doc of consumer group command
Clarify the consumer group command help message around `zookeeper`,
`bootstrap-server`, and `new-consumer` options.
Author: Vahid Hashemian <[email protected]>
Reviewers: Jason Gustafson <[email protected]>
Closes #2046 from vahidhashemian/minor/improve_consumer_group_command_doc
(cherry picked from commit 88200938f0c3ace5ec0bdb65a315ebd9b6adb4d0)
Signed-off-by: Jason Gustafson <[email protected]>
commit 7cb57dc99bc775b8a195b5ba507cb2a31a00f553
Author: Jiangjie Qin <[email protected]>
Date: 2017-05-22T00:31:31Z
KAFKA-3995; KIP-126 Allow KafkaProducer to split and resend oversized
batches
Author: Jiangjie Qin <[email protected]>
Reviewers: Joel Koshy <[email protected]>, Ismael Juma <[email protected]>
Closes #2638 from becketqin/KAFKA-3995
(cherry picked from commit 7fad45557e4cb7b345f34cec32f910b437c59bc2)
Signed-off-by: Jiangjie Qin <[email protected]>
commit 9a21bf20b623b790c9910813e373d2d61fe84c2c
Author: Jason Gustafson <[email protected]>
Date: 2017-05-24T22:26:46Z
KAFKA-5259; TransactionalId auth implies ProducerId auth
Author: Jason Gustafson <[email protected]>
Reviewers: Apurva Mehta <[email protected]>, Jun Rao <[email protected]>
Closes #3075 from hachikuji/KAFKA-5259-FIXED
(cherry picked from commit 38f6cae9e879baa35c5dbc5829bf09ecd59930c2)
Signed-off-by: Jason Gustafson <[email protected]>
commit a8dbce47f5a39ebc0fb21b7e0ef43f837b252313
Author: Jeyhun Karimov <[email protected]>
Date: 2017-05-25T02:00:37Z
KAFKA-4144 Follow-up: add one missing overload function to maintain
backward compatibility
A follow up RP to fix
[issue](https://github.com/confluentinc/examples/commit/2cd0b87bc8a7eab0e7199fa0079db6417f0e6b63#commitcomment-22200864)
Author: Jeyhun Karimov <[email protected]>
Reviewers: Matthias J. Sax, Eno Thereska, Bill Bejeck, Guozhang Wang
Closes #3109 from jeyhunkarimov/KAFKA-4144-follow-up
(cherry picked from commit c5d44af77473abb36cb9bf7ca3dead36490b8320)
Signed-off-by: Guozhang Wang <[email protected]>
commit 8dfc098ee5c5024a580dab50281e19fec2dd6cbf
Author: Jason Gustafson <[email protected]>
Date: 2017-05-25T04:00:44Z
MINOR: GroupCoordinator can append with group lock
Author: Jason Gustafson <[email protected]>
Reviewers: Guozhang Wang <[email protected]>
Closes #3133 from hachikuji/minor-replica-manager-append-refactor
(cherry picked from commit fdcee8b8b3c027cdc1b13031fa19fcfc7de5609f)
Signed-off-by: Jason Gustafson <[email protected]>
commit a89200a585593cebb4a3a1af0c2af9892babcd5b
Author: Michael André Pearce <[email protected]>
Date: 2017-05-21T21:34:19Z
KAFKA-4208; Add record headers upgrade doc
Update upgrade.html
Raising this now, as KIP-118 is pulled from release as such submitting this
without java 8 changes.
As per remaining review comment from
https://github.com/apache/kafka/pull/2772, updating the upgrade notes.
Author: Michael André Pearce <[email protected]>
Author: Michael Andre Pearce <[email protected]>
Reviewers: Jiangjie Qin <[email protected]>, Ismael Juma
<[email protected]>
Closes #2991 from michaelandrepearce/KIP-82
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
