----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28769/ -----------------------------------------------------------
(Updated Feb. 3, 2015, 6:46 p.m.) Review request for kafka. Bugs: KAFKA-1809 https://issues.apache.org/jira/browse/KAFKA-1809 Repository: kafka Description ------- KAFKA-1890 Fix bug preventing Mirror Maker from successful rebalance; reviewed by Gwen Shapira and Neha Narkhede KAFKA-1896; Record size function should check if value is null; reviewed by Guozhang Wang KAFKA-1109 Need to fix GC log configuration code, not able to override KAFKA_GC_LOG_OPTS; reviewed by Neha Narkhede KAFKA-1883 Fix NullPointerException in RequestSendThread; reviewed by Neha Narkhede KAFKA-1885 Upgrade junit dependency in core to 4.6 version to allow running individual test methods via gradle command line; reviewed by Neha Narkhede KAFKA-1818 KAFKA-1818 clean up code to more idiomatic scala usage; reviewed by Neha Narkhede and Gwen Shapira KAFKA-1902; fix MetricName so that Yammer reporter can work correctly; patched by Jun Rao; reviewed by Manikumar Reddy, Manikumar Reddy and Joel Koshy KAFKA-1861; Publishing kafka-client:test in order to utilize the helper utils in TestUtils; patched by Manikumar Reddy; reviewed by Jun Rao KAFKA-1760: New consumer. KAFKA-1760 Follow-up: fix compilation issue with Scala 2.11 first commit of refactoring. changed topicmetadata to include brokerendpoints and fixed few unit tests fixing systest and support for binding to default address fixed system tests fix default address binding and ipv6 support fix some issues regarding endpoint parsing. Also, larger segments for systest make the validation much faster added link to security wiki in doc fixing unit test after rename of ProtocolType to SecurityProtocol Following Joe's advice, added security protocol enum on client side, and modified protocol to use ID instead of string. validate producer config against enum add a second protocol for testing and modify SocketServerTests to check on multi-ports Reverted the metadata request changes and removed the explicit security protocol argument. Instead the socketserver will determine the protocol based on the port and add this to the request bump version for UpdateMetadataRequest and added support for rolling upgrades with new config following tests - fixed LeaderAndISR protocol and ZK registration for backward compatibility cleaned up some changes that were actually not necessary. hopefully making this patch slightly easier to review undoing some changes that don't belong here bring back config lost in cleanup fixes neccessary for an all non-plaintext cluster to work minor modifications following comments by Jun added missing license formatting clean up imports cleaned up V2 to not include host+port field. Using use.new.protocol flag to decide which version to serialize change endpoints collection in Broker to Map[protocol, endpoint], mostly to be clear that we intend to have one endpoint per protocol validate that listeners and advertised listeners have unique ports and protocols support legacy configs some fixes following rebase Reverted to backward compatible zk registration, changed use.new.protocol to support multiple versions and few minor fixes fixing some issues after rebase modified inter.broker.protocol config to start with security per feedback KAFKA-1729; Add constructor to javaapi to allow constructing explicitly versioned offset commit requests; reviewed by Jun Rao rebasing on trunk again Diffs (updated) ----- README.md 35e06b1cc6373f24ea6d405cccd4aacf0f458e0d bin/kafka-run-class.sh 22a9865b5939450a9d7f4ea2eee5eba2c1ec758c build.gradle 1cbab29ce83e20dae0561b51eed6fdb86d522f28 clients/src/main/java/org/apache/kafka/clients/ClientRequest.java d32c319d8ee4c46dad309ea54b136ea9798e2fd7 clients/src/main/java/org/apache/kafka/clients/ClusterConnectionStates.java 8aece7e81a804b177a6f2c12e2dc6c89c1613262 clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/ConnectionState.java ab7e3220f9b76b92ef981d695299656f041ad5ed clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 397695568d3fd8e835d8f923a89b3b00c96d0ead clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 6746275d0b2596cd6ff7ce464a3a8225ad75ef00 clients/src/main/java/org/apache/kafka/clients/NodeConnectionState.java 752a979ea0b8bde7ff6d2e1a23bf54052305d841 clients/src/main/java/org/apache/kafka/clients/consumer/CommitType.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java c0c636b3e1ba213033db6d23655032c9bbd5e378 clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 57c1807ccba9f264186f83e91f37c34b959c8060 clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerRebalanceCallback.java e4cf7d1cfa01c2844b53213a7b539cdcbcbeaa3a clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerRecord.java 16af70a5de52cca786fdea147a6a639b7dc4a311 clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerRecords.java bdf4b26942d5a8c8a9503e05908e9a9eff6228a7 clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 76efc216c9e6c3ab084461d792877092a189ad0f clients/src/main/java/org/apache/kafka/clients/consumer/MockConsumer.java fa88ac1a8b19b4294f211c4467fe68c7707ddbae clients/src/main/java/org/apache/kafka/clients/consumer/NoOffsetForPartitionException.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/consumer/OffsetMetadata.java ea423ad15eebd262d20d5ec05d592cc115229177 clients/src/main/java/org/apache/kafka/clients/consumer/internals/Heartbeat.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/consumer/internals/NoOpConsumerRebalanceCallback.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/consumer/internals/SubscriptionState.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java fc71710dd5997576d3841a1c3b0f7e19a8c9698e clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 904976fadf0610982958628eaee810b60a98d725 clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 8b3e565edd1ae04d8d34bd9f1a41e9fa8c880a75 clients/src/main/java/org/apache/kafka/clients/producer/internals/Metadata.java dcf46581b912cfb1b5c8d4cbc293d2d1444b7740 clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java 483899d2e69b33655d0e08949f5f64af2519660a clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java ccc03d8447ebba40131a70e16969686ac4aab58a clients/src/main/java/org/apache/kafka/common/Cluster.java d3299b944062d96852452de455902659ad8af757 clients/src/main/java/org/apache/kafka/common/PartitionInfo.java b15aa2c3ef2d7c4b24618ff42fd4da324237a813 clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java 98cb79b701918eca3f6ca9823b6c7b7c97b3ecec clients/src/main/java/org/apache/kafka/common/errors/ApiException.java 7c948b166a8ac07616809f260754116ae7764973 clients/src/main/java/org/apache/kafka/common/network/Selectable.java b68bbf00ab8eba6c5867d346c91188142593ca6e clients/src/main/java/org/apache/kafka/common/network/Selector.java 74d695ba39de44b6a3d15340ec0114bc4fce2ba2 clients/src/main/java/org/apache/kafka/common/protocol/ApiVersion.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/protocol/Errors.java 3316b6a1098311b8603a4a5893bf57b75d2e43cb clients/src/main/java/org/apache/kafka/common/protocol/SecurityProtocol.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/protocol/types/Struct.java 121e880a941fcd3e6392859edba11a94236494cc clients/src/main/java/org/apache/kafka/common/record/LogEntry.java e4d688cbe0c61b74ea15fc8dd3b634f9e5ee9b83 clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 040e5b91005edb8f015afdfa76fd94e0bf3cb4ca clients/src/main/java/org/apache/kafka/common/requests/ConsumerMetadataRequest.java 99b52c23d639df010bf2affc0f79d1c6e16ed67c clients/src/main/java/org/apache/kafka/common/requests/ConsumerMetadataResponse.java 8b8f591c4b2802a9cbbe34746c0b3ca4a64a8681 clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java 2fc471f64f4352eeb128bbd3941779780076fb8c clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java f719010119951e55795a583796a93a251e1ba404 clients/src/main/java/org/apache/kafka/common/requests/HeartbeatRequest.java 9512db2365d8d1f7a9a9e93b2bebabd877881143 clients/src/main/java/org/apache/kafka/common/requests/HeartbeatResponse.java 8997ffc44c18b7ae14365161c04eb423793cb8c9 clients/src/main/java/org/apache/kafka/common/requests/JoinGroupRequest.java d6e91f393179809f72d52028d6a8a2b2bb43b820 clients/src/main/java/org/apache/kafka/common/requests/JoinGroupResponse.java efe89796a7bd0d1cabc138f695b0eb28007663a7 clients/src/main/java/org/apache/kafka/common/requests/ListOffsetRequest.java 99364c1ca464f7b81be7d3da15b40ab66717a659 clients/src/main/java/org/apache/kafka/common/requests/ListOffsetResponse.java ac239712f11848755cc01399b4027ea0a8d7d8ac clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java b22ca1dce65f665d84c2a980fd82f816e93d9960 clients/src/main/java/org/apache/kafka/common/requests/MetadataResponse.java d97962d3840179b1abf01459522c58e59102ac8d clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java 3ee5cbad55ce836fd04bb954dcf6ef2f9bc3288f clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java 711232ac613786392f1389f2f883818bf4ac2a33 clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchRequest.java 90d5135b97a44d1181bd595c8eab2f2aef1a728a clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java 6b7c269ad7679df57c6bd505516075add39b7534 clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java 3dbba8a360f11b2b1db259ecce11e15d8eeaccfb clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 5220464913e6e82734e7f9ff886be7fdfa80361f clients/src/main/java/org/apache/kafka/common/serialization/Deserializer.java 3c001d33091c0f04ac3bf49a6731ab9e9f2bb0c4 clients/src/main/java/org/apache/kafka/common/utils/Utils.java 527dd0f9c47fce7310b7a37a9b95bf87f1b9c292 clients/src/test/java/org/apache/kafka/clients/MockClient.java 47b5d4ac1f2a5d162541cab658add9cc40b3aa4e clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 1a55242e9399fa4669630b55110d530f954e1279 clients/src/test/java/org/apache/kafka/clients/consumer/ConsumerExampleTest.java 29ad25e90606f065b409bc194ef9200189933aa6 clients/src/test/java/org/apache/kafka/clients/consumer/MockConsumerTest.java PRE-CREATION clients/src/test/java/org/apache/kafka/clients/consumer/internals/SubscriptionStateTest.java PRE-CREATION clients/src/test/java/org/apache/kafka/clients/producer/BufferPoolTest.java 12368038e1381f89352a5a1c98b2e0f8cbc04cbd clients/src/test/java/org/apache/kafka/clients/producer/MockProducerTest.java 3676b05eb8b83bbd8a38754fc3a979124908281b clients/src/test/java/org/apache/kafka/clients/producer/PartitionerTest.java 1d077fd4c56bf28557a0ca1e15c82cc427f36e6c clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java 66cbdf5babed3ac27c5254d945372644a053df87 clients/src/test/java/org/apache/kafka/common/config/ConfigDefTest.java 3c442a27a7ba3eaa908becc802aab6242594df85 clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 74c19573a29b0d5ac4264e1430904dca52571404 clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java a39fab532f73148316a56c0f8e9197f38ea66f79 clients/src/test/java/org/apache/kafka/test/MockSelector.java d61de529173314c044261ad9662bec735d67e97f config/server.properties 1614260b71a658b405bb24157c8f12b1f1031aa5 core/src/main/scala/kafka/admin/AdminUtils.scala 28b12c7b89a56c113b665fbde1b95f873f8624a3 core/src/main/scala/kafka/admin/TopicCommand.scala 285c0333ff43543d3e46444c1cd9374bb883bb59 core/src/main/scala/kafka/api/ConsumerMetadataRequest.scala 6d00ed090d76cd7925621a9c6db8fb00fb9d48f4 core/src/main/scala/kafka/api/ConsumerMetadataResponse.scala 84f60178f6ebae735c8aa3e79ed93fe21ac4aea7 core/src/main/scala/kafka/api/LeaderAndIsrRequest.scala 4ff7e8f8cc695551dd5d2fe65c74f6b6c571e340 core/src/main/scala/kafka/api/OffsetCommitResponse.scala 624a1c1cc540688ae2b1fb96665696a6084158e5 core/src/main/scala/kafka/api/TopicMetadata.scala 0190076df0adf906ecd332284f222ff974b315fc core/src/main/scala/kafka/api/TopicMetadataResponse.scala 92ac4e687be22e4800199c0666bfac5e0059e5bb core/src/main/scala/kafka/api/UpdateMetadataRequest.scala 530982e36b17934b8cc5fb668075a5342e142c59 core/src/main/scala/kafka/client/ClientUtils.scala ebba87f0566684c796c26cb76c64b4640a5ccfde core/src/main/scala/kafka/cluster/Broker.scala 0060add008bb3bc4b0092f2173c469fce0120be6 core/src/main/scala/kafka/cluster/BrokerEndPoint.scala PRE-CREATION core/src/main/scala/kafka/cluster/EndPoint.scala PRE-CREATION core/src/main/scala/kafka/cluster/Partition.scala b230e9a1fb1a3161f4c9d164e4890a16eceb2ad4 core/src/main/scala/kafka/cluster/SecurityProtocol.scala PRE-CREATION core/src/main/scala/kafka/common/BrokerEndPointNotAvailableException.scala PRE-CREATION core/src/main/scala/kafka/consumer/ConsumerConfig.scala 9ebbee6c16dc83767297c729d2d74ebbd063a993 core/src/main/scala/kafka/consumer/ConsumerFetcherManager.scala b9e2bea7b442a19bcebd1b350d39541a8c9dd068 core/src/main/scala/kafka/consumer/ConsumerFetcherThread.scala ee6139c901082358382c5ef892281386bf6fc91b core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 5487259751ebe19f137948249aa1fd2637d2deb4 core/src/main/scala/kafka/controller/ControllerChannelManager.scala eb492f00449744bc8d63f55b393e2a1659d38454 core/src/main/scala/kafka/controller/KafkaController.scala 66df6d2fbdbdd556da6bea0df84f93e0472c8fbf core/src/main/scala/kafka/javaapi/ConsumerMetadataResponse.scala 1b28861cdf7dfb30fc696b54f8f8f05f730f4ece core/src/main/scala/kafka/javaapi/OffsetCommitRequest.scala 873f575cc22af56d980430cde8037324eb754f80 core/src/main/scala/kafka/javaapi/OffsetCommitResponse.scala c2d3d114b82a826e5209d7e65b07c06a75ff9420 core/src/main/scala/kafka/javaapi/OffsetFetchResponse.scala 60924d2fe543d7c4b5ffe24d1355e1e57211fcc6 core/src/main/scala/kafka/javaapi/TopicMetadata.scala f384e04678df10a5b46a439f475c63371bf8e32b core/src/main/scala/kafka/log/LogConfig.scala 4631bc78106e6167ab6c0a781f40dcd8cd265598 core/src/main/scala/kafka/metrics/KafkaMetricsGroup.scala e9e49180f6de45f98e79374f519f6097b4fc8637 core/src/main/scala/kafka/network/RequestChannel.scala 7b1db3dbbb2c0676f166890f566c14aa248467ab core/src/main/scala/kafka/network/SocketServer.scala 39b1651b680b2995cedfde95d74c086d9c6219ef core/src/main/scala/kafka/producer/ProducerPool.scala 43df70bb461dd3e385e6b20396adef3c4016a3fc core/src/main/scala/kafka/server/AbstractFetcherManager.scala 20c00cb8cc2351950edbc8cb1752905a0c26e79f core/src/main/scala/kafka/server/AbstractFetcherThread.scala 8c281d4668f92eff95a4a5df3c03c4b5b20e7095 core/src/main/scala/kafka/server/KafkaApis.scala ec8d9f7ba44741db40875458bd524c4062ad6a26 core/src/main/scala/kafka/server/KafkaConfig.scala 6d74983472249eac808d361344c58cc2858ec971 core/src/main/scala/kafka/server/KafkaHealthcheck.scala 4acdd70fe9c1ee78d6510741006c2ece65450671 core/src/main/scala/kafka/server/KafkaServer.scala 89200da30a04943f0b9befe84ab17e62b747c8c4 core/src/main/scala/kafka/server/MetadataCache.scala bf81a1ab88c14be8697b441eedbeb28fa0112643 core/src/main/scala/kafka/server/ReplicaFetcherManager.scala 351dbbad3bdb709937943b336a5b0a9e0162a5e2 core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 6879e730282185bda3d6bc3659cb15af0672cecf core/src/main/scala/kafka/server/ReplicaManager.scala e58fbb922e93b0c31dff04f187fcadb4ece986d7 core/src/main/scala/kafka/tools/ConsoleConsumer.scala e455cb9a1de221a6d080e3abd909ea8f24ff7fe9 core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala d1e7c434e77859d746b8dc68dd5d5a3740425e79 core/src/main/scala/kafka/tools/ConsumerPerformance.scala 093c800ea7f8a9c972bb66e99ac4e4d431cf11cc core/src/main/scala/kafka/tools/MirrorMaker.scala 5cbc8103e33a0a234d158c048e5314e841da6249 core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala ba6ddd7a909df79a0f7d45e8b4a2af94ea0fceb6 core/src/main/scala/kafka/tools/SimpleConsumerPerformance.scala 7602b8d705970a5dab49ed36d117346a960701ac core/src/main/scala/kafka/tools/SimpleConsumerShell.scala b4f903b6c7c3bb725cac7c05eb1f885906413c4d core/src/main/scala/kafka/tools/UpdateOffsetsInZK.scala 111c9a8b94ce45d95551482e9fd3f8c1cccbf548 core/src/main/scala/kafka/utils/KafkaScheduler.scala 9a16343d2ff7192b741f0b23a6bdf58d8f2bbd3e core/src/main/scala/kafka/utils/ReplicationUtils.scala 715767380f7c284148689fd34d4bfba51abd96a0 core/src/main/scala/kafka/utils/Utils.scala 738c1af9ef5de16fdf5130daab69757a14c48b5c core/src/main/scala/kafka/utils/ZkUtils.scala c14bd455b6642f5e6eb254670bef9f57ae41d6cb core/src/test/scala/integration/kafka/api/ConsumerTest.scala PRE-CREATION core/src/test/scala/integration/kafka/api/IntegrationTestHarness.scala PRE-CREATION core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 90c0b7a19c7af8e5416e4bdba62b9824f1abd5ab core/src/test/scala/integration/kafka/api/ProducerSendTest.scala b15237b76def3b234924280fa3fdb25dbb0cc0dc core/src/test/scala/other/kafka/TestOffsetManager.scala 41f334d48897b3027ed54c58bbf4811487d3b191 core/src/test/scala/unit/kafka/KafkaConfigTest.scala 4d36b8b1173f60d43463c13c9d8c1275a84c8c28 core/src/test/scala/unit/kafka/admin/AddPartitionsTest.scala 1bf2667f47853585bc33ffb3e28256ec5f24ae84 core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala cd16ced5465d098be7a60498326b2a98c248f343 core/src/test/scala/unit/kafka/cluster/BrokerTest.scala PRE-CREATION core/src/test/scala/unit/kafka/consumer/ConsumerIteratorTest.scala c0355cc0135c6af2e346b4715659353a31723b86 core/src/test/scala/unit/kafka/integration/FetcherTest.scala 25845abbcad2e79f56f729e59239b738d3ddbc9d core/src/test/scala/unit/kafka/integration/KafkaServerTestHarness.scala 3cf7c9bcd64492d05590067a8ad11d31096a8e5e core/src/test/scala/unit/kafka/integration/PrimitiveApiTest.scala a5386a03b62956bc440b40783247c8cdf7432315 core/src/test/scala/unit/kafka/integration/TopicMetadataTest.scala 35dc071b1056e775326981573c9618d8046e601d core/src/test/scala/unit/kafka/log/LogTest.scala c2dd8eb69da8c0982a0dd20231c6f8bd58eb623e core/src/test/scala/unit/kafka/network/SocketServerTest.scala 78b431f9c88cca1bc5e430ffd41083d0e92b7e75 core/src/test/scala/unit/kafka/producer/AsyncProducerTest.scala 1db6ac329f7b54e600802c8a623f80d159d4e69b core/src/test/scala/unit/kafka/producer/SyncProducerTest.scala d60d8e0f49443f4dc8bc2cad6e2f951eda28f5cb core/src/test/scala/unit/kafka/server/AdvertiseBrokerTest.scala f0c4a56b61b4f081cf4bee799c6e9c523ff45e19 core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 82dce80d553957d8b5776a9e140c346d4e07f766 core/src/test/scala/unit/kafka/server/LeaderElectionTest.scala c2ba07c5fdbaf0e65ca033b2e4d88f45a8a15b2e core/src/test/scala/unit/kafka/server/LogOffsetTest.scala c06ee756bf0fe07e5d3c92823a476c960b37afd6 core/src/test/scala/unit/kafka/utils/MockScheduler.scala d5896ed4d3b73aecb652436b5dfc80c2835af595 core/src/test/scala/unit/kafka/utils/ReplicationUtilsTest.scala 84e08557de5acdcf0a98b192feac72836ea359b8 core/src/test/scala/unit/kafka/utils/SchedulerTest.scala b364ac2d6d623f6e86805710ca68dc32d92558c1 core/src/test/scala/unit/kafka/utils/TestUtils.scala ac15d34425795d5be20c51b01fa1108bdcd66583 system_test/replication_testsuite/testcase_1/testcase_1_properties.json 0c6d7a316cc6b51ac0755ca03558507db0706c31 system_test/utils/kafka_system_test_utils.py 41d511cbc310fa87e0f2cd2f772e479e8e3ae4e2 Diff: https://reviews.apache.org/r/28769/diff/ Testing ------- Thanks, Gwen Shapira