回复: [DISCUSS] KIP-487: Automatic Topic Creation on Producer
Hi Boyang, Justin, " 1. Could we include a link to KIP-464 and explain its relation to KIP-487? It's very hard to read through the proposal when readers only have a reference number to some KIP that is not briefed. " For KIP-464 https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113708722, Its motivation is to make default `num.partitions` and `default.replication.factor` available to AdminClient APIs. For KIP-487 https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer, I think its motivation is to make auto-create-topic configurable from producer side. "2. The KIP suggests, " In the producer, auto-creation of a topic will occur through a specific request rather than through a side effect of requesting metadata." Could we be specific such as whether we are going to introduce a separate RPC, or just send another CreateTopicRequest?" I think there is no need to introduce a separate RPC. MetadataRequest already has field allowAutoTopicCreation. For current ProducerMetadata, it sets allowAutoTopicCreation to true as https://github.com/apache/kafka/blob/448e7d7f0f46db1eae14d4fe7a1d25b7af894b09/clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerMetadata.java#L59. When broker received a metadata request and the topic doesn't exist, it will decide whether to create topic according to "allowAutoTopicCreation && config.autoCreateTopicsEnable" https://github.com/apache/kafka/blob/9a4f00f78bf37041006ae8b6432d194f603ac6cc/core/src/main/scala/kafka/server/KafkaApis.scala#L1107 So the way I implemented it is different from Justine's. Justine's PR: https://github.com/apache/kafka/pull/7075. It was implemented it by CreateTopicsRequest Jiamei's PR : https://github.com/apache/kafka/pull/8831. I added field " allowAutoTopicCreation" to ProducerMetadata, pass it to MetadataRequest. builder. As boyang said, " Won't it be more natural to assume that only when both server and client agree on turning on the switch, will a topic get created?" It was the same as what I thought. -邮件原件- 发件人: Boyang Chen 发送时间: 2020年6月24日 13:12 收件人: dev 主题: Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer Hey Justin and Jiamei, I read the KIP and skimmed over the discussion. One thing I'm not fully convinced of is why we need to deprecate the server side auto topic creation logic, which seems orthogonal towards whether a client wants to create the topic or not. Won't it be more natural to assume that only when both server and client agree on turning on the switch, will a topic get created? Some clarifications would also be appreciated: 1. Could we include a link to KIP-464 and explain its relation to KIP-487? It's very hard to read through the proposal when readers only have a reference number to some KIP that is not briefed. 2. The KIP suggests, " In the producer, auto-creation of a topic will occur through a specific request rather than through a side effect of requesting metadata." Could we be specific such as whether we are going to introduce a separate RPC, or just send another CreateTopicRequest? Boyang On Wed, Jun 17, 2020 at 8:51 AM jiamei xie wrote: > Hi all > For > https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer > , It has not been updated for a long time. And I made some update, which > has been pushed to https://github.com/apache/kafka/pull/8831 > > MetadataRequest has method Builder(List topics, boolean > allowAutoTopicCreation) by which we can set whether to enable > allowAutoTopicCreation from producer. > By default, allowAutoTopicCreation on Producer is true. And only if when > the allowAutoTopicCreation of Broker and Producer are true, the topic can > be auto-created. > > Besides, the test cases are changed: > There are 4 cases for brokerAutoTopicCreationEnable and > producerAutoCreateTopicsPolicy, Check if the topic is created under these > four cases. > If brokerAutoTopicCreationEnable and producerAutoCreateTopicsPolicy > are true: assertTrue(topicCreated) > else : intercept[ExecutionException] > > Looking forward to your feedback and comments. Thanks. > > Best wishes > Jiamei Xie > > On 2019/08/12 15:50:22, Harsha Chintalapani wrote: > > On Fri, Aug 09, 2019 at 11:12 AM, Ismael Juma wrote: > > > > > Hi all, > > > > > > A few points: > > > > > > 1. I think the way backwards compatibility is being used here is not > > > correct. Any functionality that is only enabled if set via a config is > > > backwards compatible. People may disagree with the functionality or the > > > config, but it's not a backwards compatibility issue. > > > > > > > We are tal
[jira] [Created] (KAFKA-10196) Add missing '--version' option to Kafka command producer-performance
jiamei xie created KAFKA-10196: -- Summary: Add missing '--version' option to Kafka command producer-performance Key: KAFKA-10196 URL: https://issues.apache.org/jira/browse/KAFKA-10196 Project: Kafka Issue Type: Bug Components: producer , tools Reporter: jiamei xie Assignee: jiamei xie Option '--version' is missing in Kafka command producer-performance -- This message was sent by Atlassian Jira (v8.3.4#803005)
[DISCUSS]: KIP-628: ConsumerPerformance's multi-thread implementation
Hi, I'd like to start discussion on KIP-628: ConsumerPerformance's multi-thread implementation. https://cwiki.apache.org/confluence/display/KAFKA/KIP-628%3A+ConsumerPerformance%27s+multi-thread+implementation It's about making option [threads] work in ConsumerPerformance whose implementation has been removed because new consumer replaced old consumer . If you have a moment please take a look Best wishes Jiamei Xie IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
回复: [DISCUSS]: KIP-628: ConsumerPerformance's multi-thread implementation
Sorry for sending this email to disturb you. I think I sent this email from the wrong place. So I sent it again using my gmail. So this email is abandoned -邮件原件- 发件人: Jiamei Xie 发送时间: 2020年6月19日 11:29 收件人: dev@kafka.apache.org 主题: [DISCUSS]: KIP-628: ConsumerPerformance's multi-thread implementation Hi, I'd like to start discussion on KIP-628: ConsumerPerformance's multi-thread implementation. https://cwiki.apache.org/confluence/display/KAFKA/KIP-628%3A+ConsumerPerformance%27s+multi-thread+implementation It's about making option [threads] work in ConsumerPerformance whose implementation has been removed because new consumer replaced old consumer . If you have a moment please take a look Best wishes Jiamei Xie IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
[DISCUSS]: KIP-628: ConsumerPerformance's multi-thread implementation
Hi, I'd like to start a discussion on KIP-628: ConsumerPerformance's multi-thread implementation. https://cwiki.apache.org/confluence/display/KAFKA/KIP-628%3A+ConsumerPerformance%27s+multi-thread+implementation It's about making option [threads] work in ConsumerPerformance whose implementation has been removed because new consumer replaced old consumer . If you have a moment please take a look Best wishes Jiamei Xie
[DISCUSS]: KIP-628: ConsumerPerformance's multi-thread implementation
Hi, I'd like to start discussion on KIP-628: ConsumerPerformance's multi-thread implementation. https://cwiki.apache.org/confluence/display/KAFKA/KIP-628%3A+ConsumerPerformance%27s+multi-thread+implementation It's about making option [threads] work in ConsumerPerformance whose implementation has been removed because new consumer replaced old consumer . If you have a moment please take a look Best wishes Jiamei Xie IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer
Hi all For https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer , It has not been updated for a long time. And I made some update, which has been pushed to https://github.com/apache/kafka/pull/8831 MetadataRequest has method Builder(List topics, boolean allowAutoTopicCreation) by which we can set whether to enable allowAutoTopicCreation from producer. By default, allowAutoTopicCreation on Producer is true. And only if when the allowAutoTopicCreation of Broker and Producer are true, the topic can be auto-created. Besides, the test cases are changed: There are 4 cases for brokerAutoTopicCreationEnable and producerAutoCreateTopicsPolicy, Check if the topic is created under these four cases. If brokerAutoTopicCreationEnable and producerAutoCreateTopicsPolicy are true: assertTrue(topicCreated) else : intercept[ExecutionException] Looking forward to your feedback and comments. Thanks. Best wishes Jiamei Xie On 2019/08/12 15:50:22, Harsha Chintalapani wrote: > On Fri, Aug 09, 2019 at 11:12 AM, Ismael Juma wrote: > > > Hi all, > > > > A few points: > > > > 1. I think the way backwards compatibility is being used here is not > > correct. Any functionality that is only enabled if set via a config is > > backwards compatible. People may disagree with the functionality or the > > config, but it's not a backwards compatibility issue. > > > > We are talking about both broker and producer as a single entity and run > by the same team/users. Allowing newer producer to create topics on a older > broker when auto.create.topics.enable set to false, breaks server side > contract that this config offered from the beginning. IMO, it clearly > isn't backward compatible. User who set auto.create.topic.enable on broker > will not be the same who will turn it on producer side . > > > > 2. It's an interesting question if auto topic creation via the producer > > should be a server driven choice or not. I can see the desire to have a > > server-driven default, but it seems like this is often application > > specific. Because the functionality is trivially available via AdminClient > > (released 2 years ago), it's not quite possible to control what > > applications do without the use of ACLs or policies today. > > > > > > > Producers & consumers are the majority of the clients in Kafka ecosystem. > Just because AdminClient shipped a while back that doesn't mean all users > adopting to it. To this day lot more users are aware of Producer & Consumer > APIs and running them in production compare to AdminClient. > > > > 3. Changing the create topics request in this way is highly unintuitive in > > my opinion and it relies on every client to pass the new field. For > > example, if librdkafka added auto create functionality by relying on their > > AdminClient, it would behave differently than what is proposed here. > > Forcing every client to implement this change when calling auto create from > > the producer specifically seems odd > > > > I am not sure why its unintuitive , protocols change. We add or upgrade the > existing protocols all the time. > > > Thanks, > Harsha > > . > > > > Ismael > > > > On Thu, Aug 8, 2019 at 11:51 AM Jun Rao wrote: > > > > Hi, Justine, > > > > Thanks for the KIP. Overall, it seems to be a good improvement. > > > > However, I think Harsha's point seems reasonable. We had > > auto.create.topics.enable config on the broker to allow admins to disable > > topic creation from the producer/consumer clients before we had the > > security feature. The need for that config is reduced with the security > > feature, but may still be present since not all places have security > > enabled. It's true that a non-secured environment is vulnerable to some > > additional attacks, but producer/consumer are the most common way for a > > user to interact with the broker. So, keeping that config for backward > > compatibility could still be useful if it's not introducing too much effort > > or extra confusion. > > > > > > Here is a one potential alternative way that I was thinking. We add a new > > field in the CreateTopicRequest to indicate whether it's from the producer > > or not. If auto.create.topics.enable is false, CreateTopicRequest from the > > producer will be rejected. We probably don't need to introduce the new > > config (which seems a bit hard to explain) in the producer. Instead, the > > new producer always uses MetadataRequest with AllowAutoTopicCreation set to > > false to get the metadata and if the me
Permission to create KIP
Hi, I'd like permission to create a KIP on the Apache Kafka Wiki. My WikiID is Jiamei.xie Best wishes Jiamei Xie IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
[DISCUSS] KIP-487: Automatic Topic Creation on Producer
Hi folks For https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer , It has not been updated for a long time. And I made some update, which has been pushed to https://github.com/apache/kafka/pull/8831 MetadataRequest has method Builder(List topics, boolean allowAutoTopicCreation) by which we can set whether to enable allowAutoTopicCreation from producer. By default, allowAutoTopicCreation on Producer is true. And only if when the allowAutoTopicCreation of Broker and Producer are true, the topic can be auto-created. Besides, the test cases are changed: There are 4 cases for brokerAutoTopicCreationEnable and producerAutoCreateTopicsPolicy, Check if the topic is created under these four cases. If brokerAutoTopicCreationEnable and producerAutoCreateTopicsPolicy are true: assertTrue(topicCreated) else : intercept[ExecutionException] Looking forward to your feedback and comments. Thanks. Best wishes Jiamei Xie IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
[jira] [Created] (KAFKA-10136) Make option threads of ConsumerPerformance work
jiamei xie created KAFKA-10136: -- Summary: Make option threads of ConsumerPerformance work Key: KAFKA-10136 URL: https://issues.apache.org/jira/browse/KAFKA-10136 Project: Kafka Issue Type: Bug Reporter: jiamei xie Assignee: jiamei xie Make option threads of ConsumerPerformance work -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-10126) Remove unused options in ConsumerPerformance
jiamei xie created KAFKA-10126: -- Summary: Remove unused options in ConsumerPerformance Key: KAFKA-10126 URL: https://issues.apache.org/jira/browse/KAFKA-10126 Project: Kafka Issue Type: Bug Reporter: jiamei xie Assignee: jiamei xie Option numThreadsOpt and numFetchersOpt are unused in ConsumerPerformance. It's a waste of time to test performance vs threads number. So removing it is needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-10124) ConsumerPerformance output wrong rebalance.time.ms
jiamei xie created KAFKA-10124: -- Summary: ConsumerPerformance output wrong rebalance.time.ms Key: KAFKA-10124 URL: https://issues.apache.org/jira/browse/KAFKA-10124 Project: Kafka Issue Type: Bug Components: clients, consumer Reporter: jiamei xie Assignee: jiamei xie When running consumer performance benchmark, negative fetch.time.ms and fetch.MB.sec, fetch.nMsg.sec are got, which must be wrong. bin/kafka-consumer-perf-test.sh --topic test1 --bootstrap-server localhost:9092 --messages 10 start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec 2020-06-07 05:08:52:393, 2020-06-07 05:09:46:815, 19073.6132, 350.4762, 2133, 367500.8820, 1591477733263, -1591477678841, -0., -0.0126 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-10031) Exist client create wrong topic automatically for a total-fresh restarted Kafka cluster
jiamei xie created KAFKA-10031: -- Summary: Exist client create wrong topic automatically for a total-fresh restarted Kafka cluster Key: KAFKA-10031 URL: https://issues.apache.org/jira/browse/KAFKA-10031 Project: Kafka Issue Type: Bug Components: clients Reporter: jiamei xie Assignee: jiamei xie Kill all zookeeper and kafka process. Clear zookeeper and kafka data dir. Restart zookeeper and kafka. If there are any active client. Topic used by client will be auto-created. How to reproduce? 1. Start zookeeper and kafka zookeeper and kafka config file. nohup bin/zookeeper-server-start.sh config/zookeeper.properties & nohup bin/kafka-server-start.sh config/server.properties & 2. Create topic test with 2 partitions bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic test --partitions 2 --replication-factor 1 3. Produce some data to topic test bin/kafka-producer-perf-test.sh --topic test --num-records 5000 --record-size 100 --throughput=-1 --producer-props bootstrap.servers=localhost:9092 4. Kill zookeeper and kafka. ProducerPerformance is still running. jps 21072 QuorumPeerMain 21704 ProducerPerformance 21230 Kafka 21854 Jps kill -9 21072 21230 5. Remove Zookeeper and Kafka data rm -rf /tmp/zookeeper/ rm -rf /tmp/kafka-logs/ 6. Start zookeeper and kafka nohup bin/zookeeper-server-start.sh config/zookeeper.properties & nohup bin/kafka-server-start.sh config/server.properties & 7. Check topic and you’ll see there is topic named test with partition 1. And the ProducerPerformance process continues to run normally. bin/kafka-topics.sh --describe --zookeeper localhost:2181 Topic: test PartitionCount: 1 ReplicationFactor: 1Configs: Topic: test Partition: 0Leader: 0 Replicas: 0 Isr: 0 Some output of ProducerPerformance process. 1995632 records sent, 399126.4 records/sec (38.06 MB/sec), 378.6 ms avg latency, 435.0 ms max latency. org.apache.kafka.common.errors.TimeoutException: Expiring 148 record(s) for test-1:12 ms has passed since batch creation ….. org.apache.kafka.common.errors.TimeoutException: Expiring 148 record(s) for test-1:121774 ms has passed since batch creation 1711254 records sent, 342250.8 records/sec (32.64 MB/sec), 2324.5 ms avg latency, 123473.0 ms max latency. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-10018) Change sh to bash
jiamei xie created KAFKA-10018: -- Summary: Change sh to bash Key: KAFKA-10018 URL: https://issues.apache.org/jira/browse/KAFKA-10018 Project: Kafka Issue Type: Bug Components: admin Reporter: jiamei xie Assignee: jiamei xie "#!/bin/sh" is used in kafka-server-stop.sh and zookeeper-server-stop.sh. [[ is a bash-builtin and used. Modern Debian and Ubuntu systems, which symlink sh to dash by default. So "[[: not found" will occur. Change "#!/bin/sh" into "#!/bin/bash" can avoid this error. Modify and make all scripts using bash. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9979) The "--zookeeper" option can't be used with bin/kafka-configs.sh . The document should be updated
[ https://issues.apache.org/jira/browse/KAFKA-9979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiamei xie resolved KAFKA-9979. --- Resolution: Duplicate It has been fixed > The "--zookeeper" option can't be used with bin/kafka-configs.sh . The > document should be updated > - > > Key: KAFKA-9979 > URL: https://issues.apache.org/jira/browse/KAFKA-9979 > Project: Kafka > Issue Type: Bug > Components: config, documentation >Reporter: jiamei xie >Priority: Major > > Running with command " bin/kafka-configs.sh --describe --zookeeper > localhost:2181 --entity-type brokers" , the output is > Warning: --zookeeper is deprecated and will be removed in a future version of > Kafka. > Use --bootstrap-server instead to specify a broker to connect to. > Running with command " bin/kafka-configs.sh --describe --bootstrap-server > wls-x86-hp04:9092 --entity-type brokers", the output is > Dynamic configs for broker 0 are: > Dynamic configs for broker 1 are: > Dynamic configs for broker are: > The entity name for brokers must be a valid integer broker id, found: > > In document https://kafka.apache.org/documentation/#rep-throttle, it uses > "zookeeper", which should be updated -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9979) The "--zookeeper" option can't be used with bin/kafka-configs.sh . The document should be updated
jiamei xie created KAFKA-9979: - Summary: The "--zookeeper" option can't be used with bin/kafka-configs.sh . The document should be updated Key: KAFKA-9979 URL: https://issues.apache.org/jira/browse/KAFKA-9979 Project: Kafka Issue Type: Bug Components: config, documentation Reporter: jiamei xie Running with command " bin/kafka-configs.sh --describe --zookeeper localhost:2181 --entity-type brokers" , the output is Warning: --zookeeper is deprecated and will be removed in a future version of Kafka. Use --bootstrap-server instead to specify a broker to connect to. Running with command " bin/kafka-configs.sh --describe --bootstrap-server wls-x86-hp04:9092 --entity-type brokers", the output is Dynamic configs for broker 0 are: Dynamic configs for broker 1 are: Dynamic configs for broker are: The entity name for brokers must be a valid integer broker id, found: In document https://kafka.apache.org/documentation/#rep-throttle, it uses "zookeeper", which should be updated -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9940) Command "jmh.sh -h " doesn't work
jiamei xie created KAFKA-9940: - Summary: Command "jmh.sh -h " doesn't work Key: KAFKA-9940 URL: https://issues.apache.org/jira/browse/KAFKA-9940 Project: Kafka Issue Type: Bug Components: build, documentation Reporter: jiamei xie It is said "To view all options run jmh with the -h flag." in https://github.com/apache/kafka/blob/trunk/jmh-benchmarks/README.md. But it doesn't work with the following error: /jmh.sh -h running gradlew :jmh-benchmarks:clean :jmh-benchmarks:shadowJar in quiet mode If this is not your first run and there is nothing changed, you can skip this build stage by ./jmh.sh skip-build ... Building project 'core' with Scala version 2.12.11 Building project 'streams-scala' with Scala version 2.12.11 gradle build done running JMH with args [-h] Exception in thread "main" java.lang.NoSuchMethodError: joptsimple.OptionDescriptor.options()Ljava/util/Collection; at org.openjdk.jmh.runner.options.OptionFormatter.lineFor(OptionFormatter.java:62) at org.openjdk.jmh.runner.options.OptionFormatter.format(OptionFormatter.java:51) at joptsimple.OptionParser.printHelpOn(OptionParser.java:342) at joptsimple.OptionParser.printHelpOn(OptionParser.java:328) at org.openjdk.jmh.runner.options.CommandLineOptions.showHelp(CommandLineOptions.java:457) at org.openjdk.jmh.Main.main(Main.java:46) JMH benchmarks done The different version of jopt used in KAFKA and JMH is the root cause. KAFKA uses jopt 5.0.4 while JMH uses jopt 4.6. You can get the jopt change logs from http://jopt-simple.github.io/jopt-simple/changes.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9922) Update examples README
jiamei xie created KAFKA-9922: - Summary: Update examples README Key: KAFKA-9922 URL: https://issues.apache.org/jira/browse/KAFKA-9922 Project: Kafka Issue Type: Bug Components: consumer, documentation Reporter: jiamei xie Assignee: jiamei xie Class kafka.examples.SimpleConsumerDemo was removed. But the java-simple-consumer-demo.sh was not removed and README was not updated -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9901) TimeoutError: Never saw message indicating StreamsTest finished startup on ducker@ducker07
[ https://issues.apache.org/jira/browse/KAFKA-9901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiamei xie resolved KAFKA-9901. --- Resolution: Duplicate > TimeoutError: Never saw message indicating StreamsTest finished startup on > ducker@ducker07 > -- > > Key: KAFKA-9901 > URL: https://issues.apache.org/jira/browse/KAFKA-9901 > Project: Kafka > Issue Type: Bug > Components: streams, system tests > Reporter: jiamei xie >Assignee: jiamei xie >Priority: Major > > When running _DUCKTAPE_OPTIONS="--debug" > TC_PATHS="tests/kafkatest/tests/streams/streams_broker_bounce_test.py::StreamsBrokerBounceTest.test_all_brokers_bounce" > bash tests/docker/run_tests.sh | tee debug_logs.txt > It failed because of below error. > TimeoutError: Never saw message indicating StreamsTest finished startup on > ducker@ducker07 > https://github.com/apache/kafka/pull/8443 updated the constructor of > StreamsSmokeTestJobRunnerService. But it wasn't updated in > streams_broker_bounce_test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9920) Fix NetworkDegradeTest.test_rate test error
jiamei xie created KAFKA-9920: - Summary: Fix NetworkDegradeTest.test_rate test error Key: KAFKA-9920 URL: https://issues.apache.org/jira/browse/KAFKA-9920 Project: Kafka Issue Type: Bug Components: core, system tests Reporter: jiamei xie Assignee: jiamei xie The test case of kafkatest.tests.core.network_degrade_test.NetworkDegradeTest.test_rate. rate_limit_kbit=100.device_name=eth0.task_name=rate-1000-latency-50.latency_ms=50 failed. And the error log was "Expected most of the measured rates to be within an order of magnitude of target 100. This means `tc` did not limit the bandwidth as expected." It was because that the rate_limt didn't take immediately after starting. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9901) TimeoutError: Never saw message indicating StreamsTest finished startup on ducker@ducker07
jiamei xie created KAFKA-9901: - Summary: TimeoutError: Never saw message indicating StreamsTest finished startup on ducker@ducker07 Key: KAFKA-9901 URL: https://issues.apache.org/jira/browse/KAFKA-9901 Project: Kafka Issue Type: Bug Components: streams, system tests Reporter: jiamei xie Assignee: jiamei xie When running _DUCKTAPE_OPTIONS="--debug" TC_PATHS="tests/kafkatest/tests/streams/streams_broker_bounce_test.py::StreamsBrokerBounceTest.test_all_brokers_bounce" bash tests/docker/run_tests.sh | tee debug_logs.txt It failed because of below error. TimeoutError: Never saw message indicating StreamsTest finished startup on ducker@ducker07 https://github.com/apache/kafka/pull/8443 updated the constructor of StreamsSmokeTestJobRunnerService. But it wasn't updated in streams_broker_bounce_test -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9834) Add interface to set ZSTD compresson level
[ https://issues.apache.org/jira/browse/KAFKA-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiamei xie resolved KAFKA-9834. --- Resolution: Duplicate Duplicate with KAFKA-7632 > Add interface to set ZSTD compresson level > -- > > Key: KAFKA-9834 > URL: https://issues.apache.org/jira/browse/KAFKA-9834 > Project: Kafka > Issue Type: Bug > Reporter: jiamei xie > Assignee: jiamei xie >Priority: Major > > It seems kafka use zstd default compression level 3 and doesn't have support > for setting zstd compression level. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9857) Failed to build image ducker-ak-openjdk-8 on arm
jiamei xie created KAFKA-9857: - Summary: Failed to build image ducker-ak-openjdk-8 on arm Key: KAFKA-9857 URL: https://issues.apache.org/jira/browse/KAFKA-9857 Project: Kafka Issue Type: Bug Components: system tests Reporter: jiamei xie Assignee: jiamei xie It failed to build image ducker-ak-openjdk-8 on arm and below is its log. This issue is to fix it. kafka/tests/docker$ ./run_tests.sh Sending build context to Docker daemon 53.76kB Step 1/43 : ARG jdk_version=openjdk:8 Step 2/43 : FROM $jdk_version 8: Pulling from library/openjdk no matching manifest for linux/arm64/v8 in the manifest list entries docker failed + die 'ducker-ak up failed' + echo ducker-ak up failed ducker-ak up failed + exit 1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9834) Add config to set ZSTD compresson level
jiamei xie created KAFKA-9834: - Summary: Add config to set ZSTD compresson level Key: KAFKA-9834 URL: https://issues.apache.org/jira/browse/KAFKA-9834 Project: Kafka Issue Type: Bug Reporter: jiamei xie Assignee: jiamei xie It seems kafka use zstd default compression level 3 and doesn't have support for setting zstd compression level. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9804) Extract ConsumerPerform config into one file
jiamei xie created KAFKA-9804: - Summary: Extract ConsumerPerform config into one file Key: KAFKA-9804 URL: https://issues.apache.org/jira/browse/KAFKA-9804 Project: Kafka Issue Type: Bug Components: consumer Reporter: jiamei xie Assignee: jiamei xie Configs for ConsumerPerformance are in core/src/main/scala/kafka/tools/ConsumerPerformance.scala and core/src/main/scala/kafka/tools/PerfConfig.scala. Put all of configs into class ConsumerPerformance.ConsumerPerfConfig -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9703) ProducerBatch.split takes up too many resources if the bigBatch is huge
jiamei xie created KAFKA-9703:
-
Summary: ProducerBatch.split takes up too many resources if the
bigBatch is huge
Key: KAFKA-9703
URL: https://issues.apache.org/jira/browse/KAFKA-9703
Project: Kafka
Issue Type: Bug
Reporter: jiamei xie
ProducerBatch.split takes up too many resources and might cause outOfMemory
error if the bigBatch is huge. About how I found this issue is in
https://lists.apache.org/list.html?us...@kafka.apache.org:lte=1M:MESSAGE_TOO_LARGE
Following is the code which takes a lot of resources.
{code:java}
for (Record record : recordBatch) {
assert thunkIter.hasNext();
Thunk thunk = thunkIter.next();
if (batch == null)
batch = createBatchOffAccumulatorForRecord(record,
splitBatchSize);
// A newly created batch can always host the first message.
if (!batch.tryAppendForSplit(record.timestamp(), record.key(),
record.value(), record.headers(), thunk)) {
batches.add(batch);
batch = createBatchOffAccumulatorForRecord(record,
splitBatchSize);
batch.tryAppendForSplit(record.timestamp(), record.key(),
record.value(), record.headers(), thunk);
}
{code}
Refer to RecordAccumulator#tryAppend, we can call closeForRecordAppends() after
a batch is full.
{code:java}
private RecordAppendResult tryAppend(long timestamp, byte[] key, byte[]
value, Header[] headers,
Callback callback,
Deque deque, long nowMs) {
ProducerBatch last = deque.peekLast();
if (last != null) {
FutureRecordMetadata future = last.tryAppend(timestamp, key, value,
headers, callback, nowMs);
if (future == null)
last.closeForRecordAppends();
else
return new RecordAppendResult(future, deque.size() > 1 ||
last.isFull(), false, false);
}
return null;
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9700) Negative estimatedCompressionRatio leads to misjudgment about if there is no room
jiamei xie created KAFKA-9700:
-
Summary: Negative estimatedCompressionRatio leads to misjudgment
about if there is no room
Key: KAFKA-9700
URL: https://issues.apache.org/jira/browse/KAFKA-9700
Project: Kafka
Issue Type: Bug
Components: clients
Reporter: jiamei xie
* When I run the following command
bin/kafka-producer-perf-test.sh --topic test --num-records 5000
--throughput -1 --record-size 5000 --producer-props
bootstrap.servers=server04:9092 acks=1 buffer.memory=67108864 batch.size 65536
compression.type=zstd
There was a warning:
[2020-03-06 17:36:50,216] WARN [Producer clientId=producer-1] Got error produce
response in correlation id 3261 on topic-partition test-1, splitting and
retrying (2147483647 attempts left). Error: MESSAGE_TOO_LARGE
(org.apache.kafka.clients.producer.internals.Sender)
* The batch size(65536) is smaller than max.message.bytes (1048588) . So it's
not the root cause.
* I added some logs in CompressionRatioEstimator.updateEstimation and found
there were negative currentEstimation values. The following were logs I added
public static float updateEstimation(String topic, CompressionType type, float
observedRatio) {
float[] compressionRatioForTopic = getAndCreateEstimationIfAbsent(topic);
float currentEstimation = compressionRatioForTopic[type.id];
synchronized (compressionRatioForTopic) {
if (observedRatio > currentEstimation)
{
compressionRatioForTopic[type.id] = Math.max(currentEstimation
+ COMPRESSION_RATIO_DETERIORATE_STEP, observedRatio);
}
else if (observedRatio < currentEstimation) {
compressionRatioForTopic[type.id] = currentEstimation -
COMPRESSION_RATIO_IMPROVING_STEP;
log.warn("currentEstimation is {} ,
COMPRESSION_RATIO_IMPROVING_STEP is {} , compressionRatioForTopic[type.id] is
{}, type.id is {}", currentEstimation,
COMPRESSION_RATIO_IMPROVING_STEP,compressionRatioForTopic[type.id], type.id);
}
}
return compressionRatioForTopic[type.id];
}
The observedRatio is smaller than COMPRESSION_RATIO_IMPROVING_STEP in some
cases. Some I think the else if block should be changed into
else if (observedRatio < currentEstimation) {
compressionRatioForTopic[type.id] =
Math.max(currentEstimation - COMPRESSION_RATIO_IMPROVING_STEP, observedRatio);
}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9698) Wrong default max.message.bytes in document
jiamei xie created KAFKA-9698: - Summary: Wrong default max.message.bytes in document Key: KAFKA-9698 URL: https://issues.apache.org/jira/browse/KAFKA-9698 Project: Kafka Issue Type: Bug Components: documentation Affects Versions: 2.4.0 Reporter: jiamei xie The broker default for max.message.byte has been changed to 1048588 in https://issues.apache.org/jira/browse/KAFKA-4203. But the default value in http://kafka.apache.org/documentation/ is still 112. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9391) The output of kafka-producer-perf-test.sh is ambiguous
jiamei xie created KAFKA-9391: - Summary: The output of kafka-producer-perf-test.sh is ambiguous Key: KAFKA-9391 URL: https://issues.apache.org/jira/browse/KAFKA-9391 Project: Kafka Issue Type: Bug Components: tools Reporter: jiamei xie When running the following command to test producer performance, is records/sec in the last line the average records/sec? If so, maybe avg should be added before records/sec to make it more clear. /home/linux/xjm/kafka/bin/kafka-producer-perf-test.sh --num-records 500 --topic test6 --producer-props bootstrap.servers=wls-x86-hp02:9092,wls-x86-hp04:9092 acks=1 batch.size=8192 --throughput 500 --record-size 100 2861063 records sent, 572212.6 records/sec (54.57 MB/sec), 380.0 ms avg latency, 596.0 ms max latency. 500 records sent, 603354.651864 records/sec (57.54 MB/sec), 379.81 ms avg latency, 640.00 ms max latency, 396 ms 50th, 585 ms 95th, 625 ms 99th, 634 ms 99.9th. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9225) kafka fail to run on linux-aarch64
jiamei xie created KAFKA-9225: - Summary: kafka fail to run on linux-aarch64 Key: KAFKA-9225 URL: https://issues.apache.org/jira/browse/KAFKA-9225 Project: Kafka Issue Type: Bug Components: streams Reporter: jiamei xie Attachments: compat_report.html Steps to reproduce: 1. Download Kafka latest source code 2. Build it with GRALE 3. Run [stream quick start|[https://kafka.apache.org/23/documentation/streams/quickstart]] when running bin/kafka-run-class.sh org.apache.kafka.streams.examples.wordcount.WordCountDemo It crashed with the following error message: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/xjm/kafka/core/build/dependant-libs-2.12.10/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/xjm/kafka/tools/build/dependant-libs-2.12.10/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/xjm/kafka/connect/api/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/xjm/kafka/connect/transforms/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/xjm/kafka/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/xjm/kafka/connect/file/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/xjm/kafka/connect/mirror/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/xjm/kafka/connect/mirror-client/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/xjm/kafka/connect/json/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/xjm/kafka/connect/basic-auth-extension/build/dependant-libs/slf4j-log4j12-1.7.28.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] [2019-11-19 15:42:23,277] WARN The configuration 'admin.retries' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2019-11-19 15:42:23,278] WARN The configuration 'admin.retry.backoff.ms' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig) [2019-11-19 15:42:24,278] ERROR stream-client [streams-wordcount-0f3cf88b-e2c4-4fb6-b7a3-9754fad5cd48] All stream threads have died. The instance will be in error state and should be closed. (org.apach e.kafka.streams.KafkaStreams) Exception in thread "streams-wordcount-0f3cf88b-e2c4-4fb6-b7a3-9754fad5cd48-StreamThread-1" java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni1377754636857652484.so: /tmp/librocksdbjni13777546368576524 84.so: cannot open shared object file: No such file or directory (Possible cause: can't load AMD 64-bit .so on a AARCH64-bit platform) Analyze: This issue is caused by rocksdbjni-5.18.3.jar which doesn't come with aarch64 native support. Replace rocksdbjni-5.18.3.jar with rocksdbjni-6.3.6.jar from [https://mvnrepository.com/artifact/org.rocksdb/rocksdbjni/6.3.6] can fix this problem. Attached is the binary compatibility report of rocksdbjni.jar between 5.18.3 and 6.3.6. The result is 81.8%. So is it possible to upgrade rocksdbjni to 6.3.6 in upstream? Should there be any kind of tests to execute, please kindly point me. Thanks a lot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
