[jira] [Commented] (KAFKA-1212) System test exception handling does not stop background producer threads
[ https://issues.apache.org/jira/browse/KAFKA-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880370#comment-13880370 ] John Fung commented on KAFKA-1212: -- Commented in the rb. thanks. > System test exception handling does not stop background producer threads > > > Key: KAFKA-1212 > URL: https://issues.apache.org/jira/browse/KAFKA-1212 > Project: Kafka > Issue Type: Bug >Reporter: Guozhang Wang >Assignee: Guozhang Wang > Attachments: KAFKA-1212.patch, KAFKA-1212_2014-01-21_13:20:59.patch, > KAFKA-1212_2014-01-22_11:47:22.patch, KAFKA-1212_2014-01-22_11:51:46.patch > > > When exception is thrown, the system test script stops all known entities. > However, the background producer thread cannot be stopped since it does not > register its pid in the testcase environment. We need to specifically stop > them. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Comment Edited] (KAFKA-1078) Update System Test to handle controller data returned by ZK
[
https://issues.apache.org/jira/browse/KAFKA-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789350#comment-13789350
]
John Fung edited comment on KAFKA-1078 at 10/8/13 4:27 PM:
---
1. Before the patch (no controller entity id found and system test throws
exception)
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO - Found controller with entity id:
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO - stopping controller : { "brokerid":1,
"timestamp":"1380873836165", "version":1 }
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,071 - INFO -
==
2013-10-04 01:04:38,071 - INFO - Exception while running test ''
2013-10-04 01:04:38,071 - INFO -
==
Traceback (most recent call last):
File
"/home/user/kafka/system_test/replication_testsuite/replica_basic_test.py",
line 301, in runTest
kafka_system_test_utils.stop_remote_entity(self.systemTestEnv,
controllerDict["entity_id"],
self.testcaseEnv.entityBrokerParentPidDict[controllerDict["entity_id"]])
KeyError: ''
2. After the patch (controller entity id found)
2013-10-07 16:30:48,122 - INFO -
==
2013-10-07 16:30:48,122 - INFO - Found controller with entity id: 1
2013-10-07 16:30:48,122 - INFO -
==
2013-10-07 16:30:48,123 - INFO -
==
2013-10-07 16:30:48,123 - INFO - stopping controller : 1
2013-10-07 16:30:48,123 - INFO -
==
2013-10-07 16:30:48,123 - DEBUG - executing command [ssh localhost 'pid=26805;
prev_pid=""; echo $pid; while [[ "x$pid" != "x" ]]; do prev_pid=$pid; for
child in $(ps -o pid,ppid a
x | awk "{ if ( \$2 == $pid ) { print \$1 }}"); do echo $child; pid=$child;
done; if [ $prev_pid == $pid ]; then break; fi; done' 2> /dev/null
(system_test_utils)
2013-10-07 16:30:48,477 - DEBUG - terminating (SIGTERM) process id: 26805 in
host: localhost (kafka_system_test_utils)
was (Author: jfung):
1. Before the patch (no controller entity id found)
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO - Found controller with entity id:
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO - stopping controller : { "brokerid":1,
"timestamp":"1380873836165", "version":1 }
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,071 - INFO -
==
2013-10-04 01:04:38,071 - INFO - Exception while running test ''
2013-10-04 01:04:38,071 - INFO -
==
Traceback (most recent call last):
File
"/home/user/kafka/system_test/replication_testsuite/replica_basic_test.py",
line 301, in runTest
kafka_system_test_utils.stop_remote_entity(self.systemTestEnv,
controllerDict["entity_id"],
self.testcaseEnv.entityBrokerParentPidDict[controllerDict["entity_id"]])
KeyError: ''
2. After the patch (controller entity id found)
2013-10-07 16:30:48,122 - INFO -
==
2013-10-07 16:30:48,122 - INFO - Found controller with entity id: 1
2013-10-07 16:30:48,122 - INFO -
==
2013-10-07 16:30:48,123 - INFO -
==
2013-10-07 16:30:48,123 - INFO - stopping controller : 1
2013-10-07 16:30:48,123 - INFO -
==
2013-10-07 16:30:48,123 - DEBUG - executing command [ssh localhost 'pid=26805;
prev_pid=""; echo $pid; while [[ "x$pid" != "x" ]]; do prev_pid=$pid; for
child in $(ps -o pid,ppid a
x | awk "{ if ( \$2 == $pid ) { print \$1 }}"); do echo $child; pid=$child;
done; if [ $prev_pid == $pid ]; then break; fi; done' 2> /dev/null
(system_test_utils)
2013-10-07 16:30:48,477 - DEBUG - terminating (SIGTERM) process id: 26805 in
host: localhost (kafka_system_test_utils)
> Update System Test to handle controller data returned by ZK
> ---
>
> Key: KAFKA-1078
> URL: https://issues.apache.org/jira/browse/KAFKA-1078
> Project: Kafka
> Issue Type: Bug
[jira] [Commented] (KAFKA-1078) Update System Test to handle controller data returned by ZK
[
https://issues.apache.org/jira/browse/KAFKA-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789350#comment-13789350
]
John Fung commented on KAFKA-1078:
--
1. Before the patch (no controller entity id found)
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO - Found controller with entity id:
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,070 - INFO - stopping controller : { "brokerid":1,
"timestamp":"1380873836165", "version":1 }
2013-10-04 01:04:38,070 - INFO -
==
2013-10-04 01:04:38,071 - INFO -
==
2013-10-04 01:04:38,071 - INFO - Exception while running test ''
2013-10-04 01:04:38,071 - INFO -
==
Traceback (most recent call last):
File
"/home/user/kafka/system_test/replication_testsuite/replica_basic_test.py",
line 301, in runTest
kafka_system_test_utils.stop_remote_entity(self.systemTestEnv,
controllerDict["entity_id"],
self.testcaseEnv.entityBrokerParentPidDict[controllerDict["entity_id"]])
KeyError: ''
2. After the patch (controller entity id found)
2013-10-07 16:30:48,122 - INFO -
==
2013-10-07 16:30:48,122 - INFO - Found controller with entity id: 1
2013-10-07 16:30:48,122 - INFO -
==
2013-10-07 16:30:48,123 - INFO -
==
2013-10-07 16:30:48,123 - INFO - stopping controller : 1
2013-10-07 16:30:48,123 - INFO -
==
2013-10-07 16:30:48,123 - DEBUG - executing command [ssh localhost 'pid=26805;
prev_pid=""; echo $pid; while [[ "x$pid" != "x" ]]; do prev_pid=$pid; for
child in $(ps -o pid,ppid a
x | awk "{ if ( \$2 == $pid ) { print \$1 }}"); do echo $child; pid=$child;
done; if [ $prev_pid == $pid ]; then break; fi; done' 2> /dev/null
(system_test_utils)
2013-10-07 16:30:48,477 - DEBUG - terminating (SIGTERM) process id: 26805 in
host: localhost (kafka_system_test_utils)
> Update System Test to handle controller data returned by ZK
> ---
>
> Key: KAFKA-1078
> URL: https://issues.apache.org/jira/browse/KAFKA-1078
> Project: Kafka
> Issue Type: Bug
>Reporter: John Fung
>Assignee: John Fung
> Fix For: 0.8
>
> Attachments: kafka-1078.patch
>
>
> There is a change in the result returned by ZK when querying for controller:
> 2013-10-07 09:39:15,599 - DEBUG - executing command [ssh localhost
> "JAVA_HOME=/export/apps/jdk/JDK-1_6_0_27
> /home/user/kafka/bin/kafka-run-class.sh org.apache.zookeeper.ZooKeeperMain
> -server localhost:2188 get /controller 2> /dev/null | tail -1"]
> (kafka_system_test_utils)
> 1. Previously it returned : 1
> 2. It is currently returning : { "brokerid":1, "timestamp":"1381163914835",
> "version":1 }
> The impact is that System Test doesn't have the correct entity ID for
> bouncing controller. There are a few test cases that are impacted by this
> change when they are bouncing the controller.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (KAFKA-1078) Update System Test to handle controller data returned by ZK
[
https://issues.apache.org/jira/browse/KAFKA-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Fung updated KAFKA-1078:
-
Status: Patch Available (was: Open)
> Update System Test to handle controller data returned by ZK
> ---
>
> Key: KAFKA-1078
> URL: https://issues.apache.org/jira/browse/KAFKA-1078
> Project: Kafka
> Issue Type: Bug
>Reporter: John Fung
> Attachments: kafka-1078.patch
>
>
> There is a change in the result returned by ZK when querying for controller:
> 2013-10-07 09:39:15,599 - DEBUG - executing command [ssh localhost
> "JAVA_HOME=/export/apps/jdk/JDK-1_6_0_27
> /home/user/kafka/bin/kafka-run-class.sh org.apache.zookeeper.ZooKeeperMain
> -server localhost:2188 get /controller 2> /dev/null | tail -1"]
> (kafka_system_test_utils)
> 1. Previously it returned : 1
> 2. It is currently returning : { "brokerid":1, "timestamp":"1381163914835",
> "version":1 }
> The impact is that System Test doesn't have the correct entity ID for
> bouncing controller. There are a few test cases that are impacted by this
> change when they are bouncing the controller.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (KAFKA-1078) Update System Test to handle controller data returned by ZK
[
https://issues.apache.org/jira/browse/KAFKA-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Fung updated KAFKA-1078:
-
Attachment: kafka-1078.patch
Updated kafka_system_test_utils.get_controller_attributes to read the json data
of controller.
> Update System Test to handle controller data returned by ZK
> ---
>
> Key: KAFKA-1078
> URL: https://issues.apache.org/jira/browse/KAFKA-1078
> Project: Kafka
> Issue Type: Bug
>Reporter: John Fung
> Attachments: kafka-1078.patch
>
>
> There is a change in the result returned by ZK when querying for controller:
> 2013-10-07 09:39:15,599 - DEBUG - executing command [ssh localhost
> "JAVA_HOME=/export/apps/jdk/JDK-1_6_0_27
> /home/user/kafka/bin/kafka-run-class.sh org.apache.zookeeper.ZooKeeperMain
> -server localhost:2188 get /controller 2> /dev/null | tail -1"]
> (kafka_system_test_utils)
> 1. Previously it returned : 1
> 2. It is currently returning : { "brokerid":1, "timestamp":"1381163914835",
> "version":1 }
> The impact is that System Test doesn't have the correct entity ID for
> bouncing controller. There are a few test cases that are impacted by this
> change when they are bouncing the controller.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (KAFKA-1078) Update System Test to handle controller data returned by ZK
John Fung created KAFKA-1078:
Summary: Update System Test to handle controller data returned by
ZK
Key: KAFKA-1078
URL: https://issues.apache.org/jira/browse/KAFKA-1078
Project: Kafka
Issue Type: Bug
Reporter: John Fung
There is a change in the result returned by ZK when querying for controller:
2013-10-07 09:39:15,599 - DEBUG - executing command [ssh localhost
"JAVA_HOME=/export/apps/jdk/JDK-1_6_0_27
/home/user/kafka/bin/kafka-run-class.sh org.apache.zookeeper.ZooKeeperMain
-server localhost:2188 get /controller 2> /dev/null | tail -1"]
(kafka_system_test_utils)
1. Previously it returned : 1
2. It is currently returning : { "brokerid":1, "timestamp":"1381163914835",
"version":1 }
The impact is that System Test doesn't have the correct entity ID for bouncing
controller. There are a few test cases that are impacted by this change when
they are bouncing the controller.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (KAFKA-976) Order-Preserving Mirror Maker Testcase
[
https://issues.apache.org/jira/browse/KAFKA-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13710335#comment-13710335
]
John Fung commented on KAFKA-976:
-
Uploaded kafka-976-v1.patch to add testcase_5007. Please do the following to
run the test case:
1. Check out latest 0.8 branch
2. Apply KAFKA-957.v1.patch
3. Apply KAFKA-967.v2.patch
4. Apply kafka-976-v1.patch
5. Build kafka
6. Edit /config/log4j.properties to uncomment the following 2 lines:
#log4j.logger.kafka.perf=DEBUG, kafkaAppender
#log4j.logger.kafka.perf.ProducerPerformance$ProducerThread=DEBUG,
kafkaAppender
7. Edit /system_test/testcase_to_run.json to the following:
{
"MirrorMakerTest" : [
"testcase_5007"
]
}
8. Execute the test under /system_test :
$ python -u -B system_test_runner.py 2>&1 | tee system_test_output.log
> Order-Preserving Mirror Maker Testcase
> --
>
> Key: KAFKA-976
> URL: https://issues.apache.org/jira/browse/KAFKA-976
> Project: Kafka
> Issue Type: Test
>Reporter: Guozhang Wang
>Assignee: John Fung
> Attachments: kafka-976-v1.patch
>
>
> A new testcase (5007) for mirror_maker_testsuite is needed for the
> key-dependent order-preserving mirror maker, this is related to KAFKA-957.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-976) Order-Preserving Mirror Maker Testcase
[ https://issues.apache.org/jira/browse/KAFKA-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-976: Status: Patch Available (was: Open) > Order-Preserving Mirror Maker Testcase > -- > > Key: KAFKA-976 > URL: https://issues.apache.org/jira/browse/KAFKA-976 > Project: Kafka > Issue Type: Test >Reporter: Guozhang Wang >Assignee: John Fung > Attachments: kafka-976-v1.patch > > > A new testcase (5007) for mirror_maker_testsuite is needed for the > key-dependent order-preserving mirror maker, this is related to KAFKA-957. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-976) Order-Preserving Mirror Maker Testcase
[ https://issues.apache.org/jira/browse/KAFKA-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-976: Attachment: kafka-976-v1.patch > Order-Preserving Mirror Maker Testcase > -- > > Key: KAFKA-976 > URL: https://issues.apache.org/jira/browse/KAFKA-976 > Project: Kafka > Issue Type: Test >Reporter: Guozhang Wang >Assignee: John Fung > Attachments: kafka-976-v1.patch > > > A new testcase (5007) for mirror_maker_testsuite is needed for the > key-dependent order-preserving mirror maker, this is related to KAFKA-957. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-915) System Test - Mirror Maker testcase_5001 failed
[ https://issues.apache.org/jira/browse/KAFKA-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13703827#comment-13703827 ] John Fung commented on KAFKA-915: - Hi Joel, After apply kafka-915-v1.patch (which is to create topic manually before starting mirror maker), testcase_5001 passes. However, testcase_5003 & testcase_5005 are failing due to data loss. Thanks, John > System Test - Mirror Maker testcase_5001 failed > --- > > Key: KAFKA-915 > URL: https://issues.apache.org/jira/browse/KAFKA-915 > Project: Kafka > Issue Type: Bug >Reporter: John Fung >Assignee: Joel Koshy >Priority: Critical > Labels: kafka-0.8, replication-testing > Attachments: kafka-915-v1.patch, testcase_5001_debug_logs.tar.gz > > > This case passes if brokers are set to partition = 1, replicas = 1 > It fails if brokers are set to partition = 5, replicas = 3 (consistently > reproducible) > This test case is set up as shown below. > 1. Start 2 ZK as a cluster in Source > 2. Start 2 ZK as a cluster in Target > 3. Start 3 brokers as a cluster in Source (partition = 1, replicas = 1) > 4. Start 3 brokers as a cluster in Target (partition = 1, replicas = 1) > 5. Start 1 MM > 6. Start ProducerPerformance to send some data > 7. After Producer is done, start ConsoleConsumer to consume data > 8. Stop all processes and validate if there is any data loss. > 9. No failure is introduced to any process in this test > Attached a tar file which contains the logs and system test output for both > cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-915) System Test - Mirror Maker testcase_5001 failed
[ https://issues.apache.org/jira/browse/KAFKA-915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-915: Status: Open (was: Patch Available) > System Test - Mirror Maker testcase_5001 failed > --- > > Key: KAFKA-915 > URL: https://issues.apache.org/jira/browse/KAFKA-915 > Project: Kafka > Issue Type: Bug >Reporter: John Fung >Assignee: Joel Koshy >Priority: Critical > Labels: kafka-0.8, replication-testing > Attachments: kafka-915-v1.patch, testcase_5001_debug_logs.tar.gz > > > This case passes if brokers are set to partition = 1, replicas = 1 > It fails if brokers are set to partition = 5, replicas = 3 (consistently > reproducible) > This test case is set up as shown below. > 1. Start 2 ZK as a cluster in Source > 2. Start 2 ZK as a cluster in Target > 3. Start 3 brokers as a cluster in Source (partition = 1, replicas = 1) > 4. Start 3 brokers as a cluster in Target (partition = 1, replicas = 1) > 5. Start 1 MM > 6. Start ProducerPerformance to send some data > 7. After Producer is done, start ConsoleConsumer to consume data > 8. Stop all processes and validate if there is any data loss. > 9. No failure is introduced to any process in this test > Attached a tar file which contains the logs and system test output for both > cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-915) System Test - Mirror Maker testcase_5001 failed
[ https://issues.apache.org/jira/browse/KAFKA-915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-915: Attachment: kafka-915-v1.patch > System Test - Mirror Maker testcase_5001 failed > --- > > Key: KAFKA-915 > URL: https://issues.apache.org/jira/browse/KAFKA-915 > Project: Kafka > Issue Type: Bug >Reporter: John Fung >Assignee: Joel Koshy >Priority: Critical > Labels: kafka-0.8, replication-testing > Attachments: kafka-915-v1.patch, testcase_5001_debug_logs.tar.gz > > > This case passes if brokers are set to partition = 1, replicas = 1 > It fails if brokers are set to partition = 5, replicas = 3 (consistently > reproducible) > This test case is set up as shown below. > 1. Start 2 ZK as a cluster in Source > 2. Start 2 ZK as a cluster in Target > 3. Start 3 brokers as a cluster in Source (partition = 1, replicas = 1) > 4. Start 3 brokers as a cluster in Target (partition = 1, replicas = 1) > 5. Start 1 MM > 6. Start ProducerPerformance to send some data > 7. After Producer is done, start ConsoleConsumer to consume data > 8. Stop all processes and validate if there is any data loss. > 9. No failure is introduced to any process in this test > Attached a tar file which contains the logs and system test output for both > cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-915) System Test - Mirror Maker testcase_5001 failed
[ https://issues.apache.org/jira/browse/KAFKA-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13702134#comment-13702134 ] John Fung commented on KAFKA-915: - Thanks Joel for the suggestion. It's working by calling the create topic admin before starting mirror maker. kafka-915-v1.patch is uploaded for the change. > System Test - Mirror Maker testcase_5001 failed > --- > > Key: KAFKA-915 > URL: https://issues.apache.org/jira/browse/KAFKA-915 > Project: Kafka > Issue Type: Bug >Reporter: John Fung >Assignee: Joel Koshy >Priority: Critical > Labels: kafka-0.8, replication-testing > Attachments: kafka-915-v1.patch, testcase_5001_debug_logs.tar.gz > > > This case passes if brokers are set to partition = 1, replicas = 1 > It fails if brokers are set to partition = 5, replicas = 3 (consistently > reproducible) > This test case is set up as shown below. > 1. Start 2 ZK as a cluster in Source > 2. Start 2 ZK as a cluster in Target > 3. Start 3 brokers as a cluster in Source (partition = 1, replicas = 1) > 4. Start 3 brokers as a cluster in Target (partition = 1, replicas = 1) > 5. Start 1 MM > 6. Start ProducerPerformance to send some data > 7. After Producer is done, start ConsoleConsumer to consume data > 8. Stop all processes and validate if there is any data loss. > 9. No failure is introduced to any process in this test > Attached a tar file which contains the logs and system test output for both > cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-915) System Test - Mirror Maker testcase_5001 failed
[ https://issues.apache.org/jira/browse/KAFKA-915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-915: Status: Patch Available (was: Open) > System Test - Mirror Maker testcase_5001 failed > --- > > Key: KAFKA-915 > URL: https://issues.apache.org/jira/browse/KAFKA-915 > Project: Kafka > Issue Type: Bug >Reporter: John Fung >Assignee: Joel Koshy >Priority: Critical > Labels: kafka-0.8, replication-testing > Attachments: kafka-915-v1.patch, testcase_5001_debug_logs.tar.gz > > > This case passes if brokers are set to partition = 1, replicas = 1 > It fails if brokers are set to partition = 5, replicas = 3 (consistently > reproducible) > This test case is set up as shown below. > 1. Start 2 ZK as a cluster in Source > 2. Start 2 ZK as a cluster in Target > 3. Start 3 brokers as a cluster in Source (partition = 1, replicas = 1) > 4. Start 3 brokers as a cluster in Target (partition = 1, replicas = 1) > 5. Start 1 MM > 6. Start ProducerPerformance to send some data > 7. After Producer is done, start ConsoleConsumer to consume data > 8. Stop all processes and validate if there is any data loss. > 9. No failure is introduced to any process in this test > Attached a tar file which contains the logs and system test output for both > cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-729) Gzip compression codec complains about missing SnappyInputStream
[ https://issues.apache.org/jira/browse/KAFKA-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung closed KAFKA-729. --- > Gzip compression codec complains about missing SnappyInputStream > > > Key: KAFKA-729 > URL: https://issues.apache.org/jira/browse/KAFKA-729 > Project: Kafka > Issue Type: Bug >Reporter: John Fung >Priority: Critical > > $ bin/kafka-run-class.sh kafka.perf.ProducerPerformance --broker-list > localhost:9092 --topic test_1 --messages 10 --batch-size 1 > --compression-codec 1 > java.lang.NoClassDefFoundError: org/xerial/snappy/SnappyInputStream > at > kafka.message.ByteBufferMessageSet$.kafka$message$ByteBufferMessageSet$$create(ByteBufferMessageSet.scala:41) > at > kafka.message.ByteBufferMessageSet.(ByteBufferMessageSet.scala:98) > at > kafka.producer.async.DefaultEventHandler$$anonfun$4.apply(DefaultEventHandler.scala:291) > at > kafka.producer.async.DefaultEventHandler$$anonfun$4.apply(DefaultEventHandler.scala:279) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) > at scala.collection.Iterator$class.foreach(Iterator.scala:631) > at > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161) > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194) > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) > at scala.collection.mutable.HashMap.foreach(HashMap.scala:80) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:206) > at scala.collection.mutable.HashMap.map(HashMap.scala:39) > at > kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$groupMessagesToSet(DefaultEventHandler.scala:279) > at > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:102) > at > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:98) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) > at scala.collection.Iterator$class.foreach(Iterator.scala:631) > at > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161) > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194) > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) > at scala.collection.mutable.HashMap.foreach(HashMap.scala:80) > at > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:98) > at > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72) > at > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104) > at > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87) > at > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67) > at scala.collection.immutable.Stream.foreach(Stream.scala:254) > at > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66) > at > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-729) Gzip compression codec complains about missing SnappyInputStream
[ https://issues.apache.org/jira/browse/KAFKA-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung resolved KAFKA-729. - Resolution: Fixed Not an issue any more. > Gzip compression codec complains about missing SnappyInputStream > > > Key: KAFKA-729 > URL: https://issues.apache.org/jira/browse/KAFKA-729 > Project: Kafka > Issue Type: Bug >Reporter: John Fung >Priority: Critical > > $ bin/kafka-run-class.sh kafka.perf.ProducerPerformance --broker-list > localhost:9092 --topic test_1 --messages 10 --batch-size 1 > --compression-codec 1 > java.lang.NoClassDefFoundError: org/xerial/snappy/SnappyInputStream > at > kafka.message.ByteBufferMessageSet$.kafka$message$ByteBufferMessageSet$$create(ByteBufferMessageSet.scala:41) > at > kafka.message.ByteBufferMessageSet.(ByteBufferMessageSet.scala:98) > at > kafka.producer.async.DefaultEventHandler$$anonfun$4.apply(DefaultEventHandler.scala:291) > at > kafka.producer.async.DefaultEventHandler$$anonfun$4.apply(DefaultEventHandler.scala:279) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) > at scala.collection.Iterator$class.foreach(Iterator.scala:631) > at > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161) > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194) > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) > at scala.collection.mutable.HashMap.foreach(HashMap.scala:80) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:206) > at scala.collection.mutable.HashMap.map(HashMap.scala:39) > at > kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$groupMessagesToSet(DefaultEventHandler.scala:279) > at > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:102) > at > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:98) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) > at > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80) > at scala.collection.Iterator$class.foreach(Iterator.scala:631) > at > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161) > at > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194) > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39) > at scala.collection.mutable.HashMap.foreach(HashMap.scala:80) > at > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:98) > at > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72) > at > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104) > at > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87) > at > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67) > at scala.collection.immutable.Stream.foreach(Stream.scala:254) > at > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66) > at > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-792) Update multiple attributes in testcase_xxxx_properties.json
[ https://issues.apache.org/jira/browse/KAFKA-792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung closed KAFKA-792. --- > Update multiple attributes in testcase__properties.json > --- > > Key: KAFKA-792 > URL: https://issues.apache.org/jira/browse/KAFKA-792 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > Fix For: 0.8 > > Attachments: kafka-792-v1.patch > > > The following are some of the properties need to be updated in some > testcase__properties.json. These changes have been patched in local > Hudson for a while. Create this new JIRA to check in these changes. > log.segment.bytes > default.replication.factor > num.partitions -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-716) SimpleConsumerPerformance does not consume all available messages
[ https://issues.apache.org/jira/browse/KAFKA-716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung closed KAFKA-716. --- > SimpleConsumerPerformance does not consume all available messages > - > > Key: KAFKA-716 > URL: https://issues.apache.org/jira/browse/KAFKA-716 > Project: Kafka > Issue Type: Bug >Reporter: John Fung >Priority: Blocker > Labels: p2 > > To reproduce the issue: > 1. Start 1 zookeeper > 2. Start 1 broker > 3. Send some messages > 4. Start SimpleConsumerPerformance to consume messages. The only way to > consume all messages is to set the fetch-size to be greater than the log > segment file size. > 5. This output shows that SimpleConsumerPerformance consumes only 6 messages: > $ bin/kafka-run-class.sh kafka.perf.SimpleConsumerPerformance --server > kafka://host1:9092 --topic topic_001 --fetch-size 2048 --partition 0 > start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec, > data.consumed.in.nMsg, nMsg.sec > 2013-01-21 15:09:21:124, 2013-01-21 15:09:21:165, 2048, 0.0059, 0.1429, 6, > 146.3415 > 6. This output shows that ConsoleConsumer consumes all 5500 messages (same > test as the above) > $ bin/kafka-run-class.sh kafka.consumer.ConsoleConsumer --zookeeper > host2:2181 --topic topic_001 --consumer-timeout-ms 5000 --formatter > kafka.consumer.ChecksumMessageFormatter --from-beginning | grep ^checksum | > wc -l > Consumed 5500 messages > 5500 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-716) SimpleConsumerPerformance does not consume all available messages
[ https://issues.apache.org/jira/browse/KAFKA-716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung resolved KAFKA-716. - Resolution: Fixed Not an issue > SimpleConsumerPerformance does not consume all available messages > - > > Key: KAFKA-716 > URL: https://issues.apache.org/jira/browse/KAFKA-716 > Project: Kafka > Issue Type: Bug >Reporter: John Fung >Priority: Blocker > Labels: p2 > > To reproduce the issue: > 1. Start 1 zookeeper > 2. Start 1 broker > 3. Send some messages > 4. Start SimpleConsumerPerformance to consume messages. The only way to > consume all messages is to set the fetch-size to be greater than the log > segment file size. > 5. This output shows that SimpleConsumerPerformance consumes only 6 messages: > $ bin/kafka-run-class.sh kafka.perf.SimpleConsumerPerformance --server > kafka://host1:9092 --topic topic_001 --fetch-size 2048 --partition 0 > start.time, end.time, fetch.size, data.consumed.in.MB, MB.sec, > data.consumed.in.nMsg, nMsg.sec > 2013-01-21 15:09:21:124, 2013-01-21 15:09:21:165, 2048, 0.0059, 0.1429, 6, > 146.3415 > 6. This output shows that ConsoleConsumer consumes all 5500 messages (same > test as the above) > $ bin/kafka-run-class.sh kafka.consumer.ConsoleConsumer --zookeeper > host2:2181 --topic topic_001 --consumer-timeout-ms 5000 --formatter > kafka.consumer.ChecksumMessageFormatter --from-beginning | grep ^checksum | > wc -l > Consumed 5500 messages > 5500 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-819) System Test : Add validation of log segment index to offset
[ https://issues.apache.org/jira/browse/KAFKA-819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung closed KAFKA-819. --- > System Test : Add validation of log segment index to offset > --- > > Key: KAFKA-819 > URL: https://issues.apache.org/jira/browse/KAFKA-819 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: kafka-0.8, replication-testing > Fix For: 0.8 > > Attachments: kafka-819-v1.patch > > > This can be achieved by calling DumpLogSegments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-607) System Test Transient Failure (case 4011 Log Retention) - ConsoleConsumer receives less data
[ https://issues.apache.org/jira/browse/KAFKA-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung closed KAFKA-607. --- > System Test Transient Failure (case 4011 Log Retention) - ConsoleConsumer > receives less data > > > Key: KAFKA-607 > URL: https://issues.apache.org/jira/browse/KAFKA-607 > Project: Kafka > Issue Type: Bug >Reporter: John Fung > Attachments: testcase_4011_data_and_log4j.tar.gz > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-607) System Test Transient Failure (case 4011 Log Retention) - ConsoleConsumer receives less data
[ https://issues.apache.org/jira/browse/KAFKA-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung resolved KAFKA-607. - Resolution: Fixed Not an issue any more. > System Test Transient Failure (case 4011 Log Retention) - ConsoleConsumer > receives less data > > > Key: KAFKA-607 > URL: https://issues.apache.org/jira/browse/KAFKA-607 > Project: Kafka > Issue Type: Bug >Reporter: John Fung > Attachments: testcase_4011_data_and_log4j.tar.gz > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-606) System Test Transient Failure (case 0302 GC Pause) - Log segments mismatched across replicas
[ https://issues.apache.org/jira/browse/KAFKA-606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung closed KAFKA-606. --- > System Test Transient Failure (case 0302 GC Pause) - Log segments mismatched > across replicas > > > Key: KAFKA-606 > URL: https://issues.apache.org/jira/browse/KAFKA-606 > Project: Kafka > Issue Type: Bug >Reporter: John Fung > Attachments: testcase_0302_data_and_log4j.tar.gz > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-651) Create testcases on auto create topics
[ https://issues.apache.org/jira/browse/KAFKA-651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung closed KAFKA-651. --- > Create testcases on auto create topics > -- > > Key: KAFKA-651 > URL: https://issues.apache.org/jira/browse/KAFKA-651 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > Attachments: kafka-651-v1.patch, kafka-651-v2.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-606) System Test Transient Failure (case 0302 GC Pause) - Log segments mismatched across replicas
[ https://issues.apache.org/jira/browse/KAFKA-606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung resolved KAFKA-606. - Resolution: Fixed Not an issue any more. > System Test Transient Failure (case 0302 GC Pause) - Log segments mismatched > across replicas > > > Key: KAFKA-606 > URL: https://issues.apache.org/jira/browse/KAFKA-606 > Project: Kafka > Issue Type: Bug >Reporter: John Fung > Attachments: testcase_0302_data_and_log4j.tar.gz > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-609) System Test Transient Failure 9001 (Migration tool) - ConsoleConsumer terminates after "can't rebalance after 4 retries"
[ https://issues.apache.org/jira/browse/KAFKA-609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung closed KAFKA-609. --- > System Test Transient Failure 9001 (Migration tool) - ConsoleConsumer > terminates after "can't rebalance after 4 retries" > > > Key: KAFKA-609 > URL: https://issues.apache.org/jira/browse/KAFKA-609 > Project: Kafka > Issue Type: Bug >Reporter: John Fung > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-624) Add 07 ConsoleConsumer to validate message content for Migration Tool testcases
[ https://issues.apache.org/jira/browse/KAFKA-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung resolved KAFKA-624. - Resolution: Fixed This is handled in KAFKA-882 & KAFKA-883 > Add 07 ConsoleConsumer to validate message content for Migration Tool > testcases > --- > > Key: KAFKA-624 > URL: https://issues.apache.org/jira/browse/KAFKA-624 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-619) Regression : System Test 900x (Migration Tool) - java.lang.ClassCastException: kafka.message.Message cannot be cast to [B
[
https://issues.apache.org/jira/browse/KAFKA-619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Fung closed KAFKA-619.
---
> Regression : System Test 900x (Migration Tool) -
> java.lang.ClassCastException: kafka.message.Message cannot be cast to [B
> -
>
> Key: KAFKA-619
> URL: https://issues.apache.org/jira/browse/KAFKA-619
> Project: Kafka
> Issue Type: Bug
>Reporter: John Fung
>
> This error is happening in : testcase_900x (migration tool test group). The
> issue is that no data is received by ConsoleConsumer. All Migration Tool
> log4j messages are showing the following error.
> . . .
> [2012-11-16 08:28:55,361] INFO FetchRunnable-1 start fetching topic: test_1
> part: 3 offset: 0 from 127.0.0.1:9093 (kafka.consumer.FetcherRunnable)
> [2012-11-16 08:28:55,361] INFO FetchRunnable-0 start fetching topic: test_1
> part: 3 offset: 0 from 127.0.0.1:9092 (kafka.consumer.FetcherRunnable)
> [2012-11-16 08:28:55,361] INFO FetchRunnable-2 start fetching topic: test_1
> part: 0 offset: 0 from 127.0.0.1:9091 (kafka.consumer.FetcherRunnable)
> Migration thread failure due to java.lang.ClassCastException:
> kafka.message.Message cannot be cast to [B
> java.lang.ClassCastException: kafka.message.Message cannot be cast to [B
> at kafka.serializer.DefaultEncoder.toBytes(Encoder.scala:36)
> at
> kafka.producer.async.DefaultEventHandler$$anonfun$serialize$1.apply(DefaultEventHandler.scala:111)
> at
> kafka.producer.async.DefaultEventHandler$$anonfun$serialize$1.apply(DefaultEventHandler.scala:106)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
> at
> scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:32)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> at scala.collection.mutable.WrappedArray.map(WrappedArray.scala:32)
> at
> kafka.producer.async.DefaultEventHandler.serialize(DefaultEventHandler.scala:106)
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:47)
> at kafka.producer.Producer.send(Producer.scala:75)
> at kafka.javaapi.producer.Producer.send(Producer.scala:32)
> at
> kafka.tools.KafkaMigrationTool$MigrationThread.run(KafkaMigrationTool.java:287)
> [2012-11-16 08:30:04,854] INFO
> test-consumer-group_jfung-ld-1353083318174-2b62271b begin rebalancing
> consumer test-consumer-group_jfung-ld-1353083318174-2b62271b try #0
> (kafka.consumer.ZookeeperConsumerConnector)
> [2012-11-16 08:30:04,858] INFO Constructing topic count for
> test-consumer-group_jfung-ld-1353083318174-2b62271b from *2*.* using
> \*(\p{Digit}+)\*(.*) as pattern. (kafka.consumer.TopicCount$)
> . . .
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-624) Add 07 ConsoleConsumer to validate message content for Migration Tool testcases
[ https://issues.apache.org/jira/browse/KAFKA-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung closed KAFKA-624. --- > Add 07 ConsoleConsumer to validate message content for Migration Tool > testcases > --- > > Key: KAFKA-624 > URL: https://issues.apache.org/jira/browse/KAFKA-624 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-621) System Test 9051 : ConsoleConsumer doesn't receives any data for 20 topics but works for 10
[ https://issues.apache.org/jira/browse/KAFKA-621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung resolved KAFKA-621. - Resolution: Fixed Not an issue any more. > System Test 9051 : ConsoleConsumer doesn't receives any data for 20 topics > but works for 10 > --- > > Key: KAFKA-621 > URL: https://issues.apache.org/jira/browse/KAFKA-621 > Project: Kafka > Issue Type: Bug >Reporter: John Fung > Attachments: kafka-621-reproduce-issue.patch > > > * This issue may be related to KAFKA-618 > * To reproduce the issue: > 1. Download the latest 0.8 branch and apply the attached patch > 2. In , execute "./sbt update package" > 3. In /system_test, execute "python -B system_test_runner.py" > and it will execute testcase_9051 > * The validation output would be as follows: > validation_status : > Unique messages from consumer on [t001] : 0 > Unique messages from consumer on [t002] : 0 > . . . > Unique messages from consumer on [t019] : 0 > Unique messages from consumer on [t020] : 0 > Unique messages from producer on [t001] : 1000 > Unique messages from producer on [t002] : 1000 > . . . > Unique messages from producer on [t018] : 1000 > Unique messages from producer on [t019] : 1000 > Unique messages from producer on [t020] : 1000 > Validate for data matched on topic [t001] : FAILED > Validate for data matched on topic [t002] : FAILED > . . . > Validate for data matched on topic [t019] : FAILED > Validate for data matched on topic [t020] : FAILED > Validate for merged log segment checksum in cluster [source] : PASSED > * However, it will work fine if there are only 10 topics > In > system_test/replication_testsuite/testcase_9051/testcase_9051_properties.json, > update the following line to 10 topics: > "topic": > "t001,t002,t003,t004,t005,t006,t007,t008,t009,t010,t011,t012,t013,t014,t015,t016,t017,t018,t019,t020", -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-621) System Test 9051 : ConsoleConsumer doesn't receives any data for 20 topics but works for 10
[ https://issues.apache.org/jira/browse/KAFKA-621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung closed KAFKA-621. --- > System Test 9051 : ConsoleConsumer doesn't receives any data for 20 topics > but works for 10 > --- > > Key: KAFKA-621 > URL: https://issues.apache.org/jira/browse/KAFKA-621 > Project: Kafka > Issue Type: Bug >Reporter: John Fung > Attachments: kafka-621-reproduce-issue.patch > > > * This issue may be related to KAFKA-618 > * To reproduce the issue: > 1. Download the latest 0.8 branch and apply the attached patch > 2. In , execute "./sbt update package" > 3. In /system_test, execute "python -B system_test_runner.py" > and it will execute testcase_9051 > * The validation output would be as follows: > validation_status : > Unique messages from consumer on [t001] : 0 > Unique messages from consumer on [t002] : 0 > . . . > Unique messages from consumer on [t019] : 0 > Unique messages from consumer on [t020] : 0 > Unique messages from producer on [t001] : 1000 > Unique messages from producer on [t002] : 1000 > . . . > Unique messages from producer on [t018] : 1000 > Unique messages from producer on [t019] : 1000 > Unique messages from producer on [t020] : 1000 > Validate for data matched on topic [t001] : FAILED > Validate for data matched on topic [t002] : FAILED > . . . > Validate for data matched on topic [t019] : FAILED > Validate for data matched on topic [t020] : FAILED > Validate for merged log segment checksum in cluster [source] : PASSED > * However, it will work fine if there are only 10 topics > In > system_test/replication_testsuite/testcase_9051/testcase_9051_properties.json, > update the following line to 10 topics: > "topic": > "t001,t002,t003,t004,t005,t006,t007,t008,t009,t010,t011,t012,t013,t014,t015,t016,t017,t018,t019,t020", -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-883) System Test - update migration tool testsuite after 0.7 ProducerPerformance sends seq MessageID
[ https://issues.apache.org/jira/browse/KAFKA-883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698140#comment-13698140 ] John Fung commented on KAFKA-883: - This patch is OK to check in ONLY after the following : 1. Build a new kafka-perf-0.7.0.jar from KAFKA-882 2. Check in the above new kafka-perf-0.7.0.jar into system_test/migration_tool_testsuite/0.7/lib to replace the existing one. > System Test - update migration tool testsuite after 0.7 ProducerPerformance > sends seq MessageID > --- > > Key: KAFKA-883 > URL: https://issues.apache.org/jira/browse/KAFKA-883 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: kafka-0.8, replication-testing > Attachments: kafka-883-v1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-883) System Test - update migration tool testsuite after 0.7 ProducerPerformance sends seq MessageID
[ https://issues.apache.org/jira/browse/KAFKA-883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-883: Status: Patch Available (was: Open) > System Test - update migration tool testsuite after 0.7 ProducerPerformance > sends seq MessageID > --- > > Key: KAFKA-883 > URL: https://issues.apache.org/jira/browse/KAFKA-883 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: kafka-0.8, replication-testing > Attachments: kafka-883-v1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-882) Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8
[ https://issues.apache.org/jira/browse/KAFKA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698128#comment-13698128 ] John Fung commented on KAFKA-882: - ProducerPerformance in 0.7 has quite a few nested if-else conditional logics in addition to while & for loops. In order to keep it simple, this patch contains the change to send sequential MessageID only when the following arguments are specified: --initial-message-id --vary-message-size --async Otherwise, it will preserve the original 0.7 behavior. > Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8 > -- > > Key: KAFKA-882 > URL: https://issues.apache.org/jira/browse/KAFKA-882 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: kafka > Attachments: kafka-882-v1.patch > > > Also update related files in the following in Kafka 0.8: > 1. system_test/migration_tool_testsuite/config > 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after > this JIRA is completed) > 3. > system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-883) System Test - update migration tool testsuite after 0.7 ProducerPerformance sends seq MessageID
[ https://issues.apache.org/jira/browse/KAFKA-883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-883: Attachment: kafka-883-v1.patch uploaded kafka-883-v1.patch > System Test - update migration tool testsuite after 0.7 ProducerPerformance > sends seq MessageID > --- > > Key: KAFKA-883 > URL: https://issues.apache.org/jira/browse/KAFKA-883 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: kafka-0.8, replication-testing > Attachments: kafka-883-v1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-882) Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8
[ https://issues.apache.org/jira/browse/KAFKA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-882: Status: Patch Available (was: Open) > Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8 > -- > > Key: KAFKA-882 > URL: https://issues.apache.org/jira/browse/KAFKA-882 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: kafka > Attachments: kafka-882-v1.patch > > > Also update related files in the following in Kafka 0.8: > 1. system_test/migration_tool_testsuite/config > 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after > this JIRA is completed) > 3. > system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work stopped] (KAFKA-882) Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8
[ https://issues.apache.org/jira/browse/KAFKA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-882 stopped by John Fung. > Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8 > -- > > Key: KAFKA-882 > URL: https://issues.apache.org/jira/browse/KAFKA-882 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: kafka > Attachments: kafka-882-v1.patch > > > Also update related files in the following in Kafka 0.8: > 1. system_test/migration_tool_testsuite/config > 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after > this JIRA is completed) > 3. > system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-882) Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8
[ https://issues.apache.org/jira/browse/KAFKA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-882: Attachment: kafka-882-v1.patch uploaded kafka-882-v1.patch > Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8 > -- > > Key: KAFKA-882 > URL: https://issues.apache.org/jira/browse/KAFKA-882 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: kafka > Attachments: kafka-882-v1.patch > > > Also update related files in the following in Kafka 0.8: > 1. system_test/migration_tool_testsuite/config > 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after > this JIRA is completed) > 3. > system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (KAFKA-882) Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8
[ https://issues.apache.org/jira/browse/KAFKA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-882 started by John Fung. > Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8 > -- > > Key: KAFKA-882 > URL: https://issues.apache.org/jira/browse/KAFKA-882 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: kafka > > Also update related files in the following in Kafka 0.8: > 1. system_test/migration_tool_testsuite/config > 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after > this JIRA is completed) > 3. > system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-879) In system test, read the new leader from zookeeper instead of broker log on completion of become-leader state transition
[ https://issues.apache.org/jira/browse/KAFKA-879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-879: Status: Patch Available (was: Open) kafka-879-v1.patch contains the change to query zookeeper to find out the leader change. However, the info provided by zookeeper does not include the timestamp detailed enough to determine the leader election latency. Therefore, the leader election latency validation is disabled in the patch. Please note that the leader election latency retrieved from the log messages is not accurate though. > In system test, read the new leader from zookeeper instead of broker log on > completion of become-leader state transition > > > Key: KAFKA-879 > URL: https://issues.apache.org/jira/browse/KAFKA-879 > Project: Kafka > Issue Type: Bug >Reporter: Swapnil Ghike >Assignee: John Fung > Labels: bugs > Attachments: kafka-879-v1.patch > > > Currently the system test reads the new leader from a broker log statement on > completion of become-leader state transition. The log statements can change > with time and can also move around. > We can read the leader info from zookeeper given a topic, partition. The path > is #2 at > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper. > The Zookeeper data structures are unlikely to change in foreseeable future. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-879) In system test, read the new leader from zookeeper instead of broker log on completion of become-leader state transition
[ https://issues.apache.org/jira/browse/KAFKA-879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-879: Attachment: kafka-879-v1.patch Uploaded kafka-879-v1.patch > In system test, read the new leader from zookeeper instead of broker log on > completion of become-leader state transition > > > Key: KAFKA-879 > URL: https://issues.apache.org/jira/browse/KAFKA-879 > Project: Kafka > Issue Type: Bug >Reporter: Swapnil Ghike >Assignee: John Fung > Labels: bugs > Attachments: kafka-879-v1.patch > > > Currently the system test reads the new leader from a broker log statement on > completion of become-leader state transition. The log statements can change > with time and can also move around. > We can read the leader info from zookeeper given a topic, partition. The path > is #2 at > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper. > The Zookeeper data structures are unlikely to change in foreseeable future. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-752) Enhance 07 ProducerPerformance to print payload checksum
[ https://issues.apache.org/jira/browse/KAFKA-752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung resolved KAFKA-752. - Resolution: Fixed > Enhance 07 ProducerPerformance to print payload checksum > > > Key: KAFKA-752 > URL: https://issues.apache.org/jira/browse/KAFKA-752 > Project: Kafka > Issue Type: Task >Reporter: John Fung > > Currently 07 ProducerPerformance can only print message.checksum. In > Migration tool test cases, it is necessary to validate that the payload > checksums of 07 ProducerPerformance and 08 ConsoleConsumer are matching. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-932) System Test - set retry.backoff.ms=300 to all test cases
[ https://issues.apache.org/jira/browse/KAFKA-932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-932: Status: Patch Available (was: Open) > System Test - set retry.backoff.ms=300 to all test cases > > > Key: KAFKA-932 > URL: https://issues.apache.org/jira/browse/KAFKA-932 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-932-v1.patch > > > 1. Set retry.backoff.ms=300 to all test cases > 2. Set each bouncing 1 minute apart in replication_testsuite -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-932) System Test - set retry.backoff.ms=300 to all test cases
[ https://issues.apache.org/jira/browse/KAFKA-932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-932: Attachment: kafka-932-v1.patch Uploaded kafka-932-v1.patch which has been tested in a local Hudson build. > System Test - set retry.backoff.ms=300 to all test cases > > > Key: KAFKA-932 > URL: https://issues.apache.org/jira/browse/KAFKA-932 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-932-v1.patch > > > 1. Set retry.backoff.ms=300 to all test cases > 2. Set each bouncing 1 minute apart in replication_testsuite -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-932) System Test - set retry.backoff.ms=300 to all test cases
John Fung created KAFKA-932: --- Summary: System Test - set retry.backoff.ms=300 to all test cases Key: KAFKA-932 URL: https://issues.apache.org/jira/browse/KAFKA-932 Project: Kafka Issue Type: Task Reporter: John Fung 1. Set retry.backoff.ms=300 to all test cases 2. Set each bouncing 1 minute apart in replication_testsuite -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-922) System Test - set retry.backoff.ms=300 to testcase_0119
[ https://issues.apache.org/jira/browse/KAFKA-922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-922: Attachment: kafka-922-v1.patch Uploaded kafka-922-v2.patch. > System Test - set retry.backoff.ms=300 to testcase_0119 > --- > > Key: KAFKA-922 > URL: https://issues.apache.org/jira/browse/KAFKA-922 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-922-v1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-922) System Test - set retry.backoff.ms=300 to testcase_0119
[ https://issues.apache.org/jira/browse/KAFKA-922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-922: Status: Patch Available (was: Open) Trivial patch to set retry.backoff.ms=300 > System Test - set retry.backoff.ms=300 to testcase_0119 > --- > > Key: KAFKA-922 > URL: https://issues.apache.org/jira/browse/KAFKA-922 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-922-v1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-922) System Test - set retry.backoff.ms=300 to testcase_0119
John Fung created KAFKA-922: --- Summary: System Test - set retry.backoff.ms=300 to testcase_0119 Key: KAFKA-922 URL: https://issues.apache.org/jira/browse/KAFKA-922 Project: Kafka Issue Type: Task Reporter: John Fung -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-915) System Test - Mirror Maker testcase_5001 failed
[ https://issues.apache.org/jira/browse/KAFKA-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664653#comment-13664653 ] John Fung commented on KAFKA-915: - This issue is not reproducible in the following commit (or before it) : commit 988d4d8e65a14390abd748318a64e281e4a37c19 Author: Neha Narkhede Date: Tue Apr 30 17:20:54 2013 -0700 > System Test - Mirror Maker testcase_5001 failed > --- > > Key: KAFKA-915 > URL: https://issues.apache.org/jira/browse/KAFKA-915 > Project: Kafka > Issue Type: Bug >Reporter: John Fung >Priority: Critical > Labels: kafka-0.8, replication-testing > Attachments: testcase_5001_debug_logs.tar.gz > > > This case passes if brokers are set to partition = 1, replicas = 1 > It fails if brokers are set to partition = 5, replicas = 3 (consistently > reproducible) > This test case is set up as shown below. > 1. Start 2 ZK as a cluster in Source > 2. Start 2 ZK as a cluster in Target > 3. Start 3 brokers as a cluster in Source (partition = 1, replicas = 1) > 4. Start 3 brokers as a cluster in Target (partition = 1, replicas = 1) > 5. Start 1 MM > 6. Start ProducerPerformance to send some data > 7. After Producer is done, start ConsoleConsumer to consume data > 8. Stop all processes and validate if there is any data loss. > 9. No failure is introduced to any process in this test > Attached a tar file which contains the logs and system test output for both > cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-915) System Test - Mirror Maker testcase_5001 failed
[ https://issues.apache.org/jira/browse/KAFKA-915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-915: Attachment: testcase_5001_debug_logs.tar.gz > System Test - Mirror Maker testcase_5001 failed > --- > > Key: KAFKA-915 > URL: https://issues.apache.org/jira/browse/KAFKA-915 > Project: Kafka > Issue Type: Bug >Reporter: John Fung >Priority: Critical > Labels: kafka-0.8, replication-testing > Attachments: testcase_5001_debug_logs.tar.gz > > > This case passes if brokers are set to partition = 1, replicas = 1 > It fails if brokers are set to partition = 5, replicas = 3 (consistently > reproducible) > This test case is set up as shown below. > 1. Start 2 ZK as a cluster in Source > 2. Start 2 ZK as a cluster in Target > 3. Start 3 brokers as a cluster in Source (partition = 1, replicas = 1) > 4. Start 3 brokers as a cluster in Target (partition = 1, replicas = 1) > 5. Start 1 MM > 6. Start ProducerPerformance to send some data > 7. After Producer is done, start ConsoleConsumer to consume data > 8. Stop all processes and validate if there is any data loss. > 9. No failure is introduced to any process in this test > Attached a tar file which contains the logs and system test output for both > cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-915) System Test - Mirror Maker testcase_5001 failed
John Fung created KAFKA-915: --- Summary: System Test - Mirror Maker testcase_5001 failed Key: KAFKA-915 URL: https://issues.apache.org/jira/browse/KAFKA-915 Project: Kafka Issue Type: Bug Reporter: John Fung Priority: Critical This case passes if brokers are set to partition = 1, replicas = 1 It fails if brokers are set to partition = 5, replicas = 3 (consistently reproducible) This test case is set up as shown below. 1. Start 2 ZK as a cluster in Source 2. Start 2 ZK as a cluster in Target 3. Start 3 brokers as a cluster in Source (partition = 1, replicas = 1) 4. Start 3 brokers as a cluster in Target (partition = 1, replicas = 1) 5. Start 1 MM 6. Start ProducerPerformance to send some data 7. After Producer is done, start ConsoleConsumer to consume data 8. Stop all processes and validate if there is any data loss. 9. No failure is introduced to any process in this test Attached a tar file which contains the logs and system test output for both cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-868) System Test - add test case for rolling controlled shutdown
[ https://issues.apache.org/jira/browse/KAFKA-868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-868: Attachment: kafka-868-v3.patch Uploaded kafka-868-v3.patch after latest 0.8 branch rebase > System Test - add test case for rolling controlled shutdown > --- > > Key: KAFKA-868 > URL: https://issues.apache.org/jira/browse/KAFKA-868 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Labels: kafka-0.8, replication-testing > Attachments: kafka-868-v1.patch, kafka-868-v2.patch, > kafka-868-v3.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-883) System Test - update migration tool testsuite after 0.7 ProducerPerformance sends seq MessageID
[ https://issues.apache.org/jira/browse/KAFKA-883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13643018#comment-13643018 ] John Fung commented on KAFKA-883: - Update the following after KAFKA-882 is completed: 1. system_test/migration_tool_testsuite/config 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after this JIRA is completed) 3. system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json > System Test - update migration tool testsuite after 0.7 ProducerPerformance > sends seq MessageID > --- > > Key: KAFKA-883 > URL: https://issues.apache.org/jira/browse/KAFKA-883 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: kafka-0.8, replication-testing > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-883) System Test - update migration tool testsuite after 0.7 ProducerPerformance sends seq MessageID
John Fung created KAFKA-883: --- Summary: System Test - update migration tool testsuite after 0.7 ProducerPerformance sends seq MessageID Key: KAFKA-883 URL: https://issues.apache.org/jira/browse/KAFKA-883 Project: Kafka Issue Type: Task Reporter: John Fung Assignee: John Fung -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-882) Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8
[ https://issues.apache.org/jira/browse/KAFKA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-882: Description: Also update related files in the following in Kafka 0.8: 1. system_test/migration_tool_testsuite/config 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after this JIRA is completed) 3. system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json was: Also update related files in the following : 1. system_test/migration_tool_testsuite/config 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after this JIRA is completed) 3. system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json > Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8 > -- > > Key: KAFKA-882 > URL: https://issues.apache.org/jira/browse/KAFKA-882 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: kafka > > Also update related files in the following in Kafka 0.8: > 1. system_test/migration_tool_testsuite/config > 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after > this JIRA is completed) > 3. > system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-882) Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8
[ https://issues.apache.org/jira/browse/KAFKA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-882: Labels: kafka (was: ) > Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8 > -- > > Key: KAFKA-882 > URL: https://issues.apache.org/jira/browse/KAFKA-882 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: kafka > > Also update related files in the following : > 1. system_test/migration_tool_testsuite/config > 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after > this JIRA is completed) > 3. > system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-882) Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8
John Fung created KAFKA-882: --- Summary: Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8 Key: KAFKA-882 URL: https://issues.apache.org/jira/browse/KAFKA-882 Project: Kafka Issue Type: Task Reporter: John Fung Assignee: John Fung Also update related files in the following : 1. system_test/migration_tool_testsuite/config 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after this JIRA is completed) 3. system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-871) Rename ZkConfig properties
[ https://issues.apache.org/jira/browse/KAFKA-871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13641966#comment-13641966 ] John Fung commented on KAFKA-871: - [~swapnilghike] : The test result is showing 0 messages for both producer & consumer. There may be some settings which are configured incorrectly. [~junrao] : The broker.list property line is also used in the Hudson nightly job. > Rename ZkConfig properties > -- > > Key: KAFKA-871 > URL: https://issues.apache.org/jira/browse/KAFKA-871 > Project: Kafka > Issue Type: Bug >Reporter: Swapnil Ghike >Assignee: Swapnil Ghike >Priority: Blocker > Labels: kafka-0.8, p1 > Attachments: kafka-871-v1.patch, kafka-871-v2.patch, > kafka-871-v3.patch > > > For clarity. Renaming these properties should help in migration from 0.7 to > 0.8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-871) Rename ZkConfig properties
[ https://issues.apache.org/jira/browse/KAFKA-871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13640711#comment-13640711 ] John Fung commented on KAFKA-871: - The following are older system testing scripts in bash before system test is ported to Python. It is probably safe to be removed from the branch to avoid confusion. 1. system_test/broker_failure 2. system_test/mirror_maker (NOT mirror_maker_testsuite) 3. system_test/producer_perf 4. system_test/common > Rename ZkConfig properties > -- > > Key: KAFKA-871 > URL: https://issues.apache.org/jira/browse/KAFKA-871 > Project: Kafka > Issue Type: Bug >Reporter: Swapnil Ghike >Assignee: Swapnil Ghike >Priority: Blocker > Labels: kafka-0.8, p1 > Attachments: kafka-871-v1.patch, kafka-871-v2.patch > > > For clarity. Renaming these properties should help in migration from 0.7 to > 0.8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-871) Rename ZkConfig properties
[ https://issues.apache.org/jira/browse/KAFKA-871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13639382#comment-13639382 ] John Fung commented on KAFKA-871: - Hi Swapnil, In system_test/migration_tool_testsuite, the source cluster are running in 0.7. It is necessary to distinguish between 0.7 & 0.8 config in the testcase__properties.json file. Let's walk through the changes needed. > Rename ZkConfig properties > -- > > Key: KAFKA-871 > URL: https://issues.apache.org/jira/browse/KAFKA-871 > Project: Kafka > Issue Type: Bug >Reporter: Swapnil Ghike >Assignee: Swapnil Ghike >Priority: Blocker > Labels: kafka-0.8, p1 > Attachments: kafka-871-v1.patch > > > For clarity. Renaming these properties should help in migration from 0.7 to > 0.8. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-868) System Test - add test case for rolling controlled shutdown
[ https://issues.apache.org/jira/browse/KAFKA-868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-868: Attachment: kafka-868-v2.patch Uploaded kafka-868-v2.patch with the following change : At the end of each iteration, rename producer & consumer logs instead of deleting the files for future troubleshooting. > System Test - add test case for rolling controlled shutdown > --- > > Key: KAFKA-868 > URL: https://issues.apache.org/jira/browse/KAFKA-868 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Labels: kafka-0.8, replication-testing > Attachments: kafka-868-v1.patch, kafka-868-v2.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-868) System Test - add test case for rolling controlled shutdown
[ https://issues.apache.org/jira/browse/KAFKA-868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-868: Status: Patch Available (was: Open) > System Test - add test case for rolling controlled shutdown > --- > > Key: KAFKA-868 > URL: https://issues.apache.org/jira/browse/KAFKA-868 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Labels: kafka-0.8, replication-testing > Attachments: kafka-868-v1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-868) System Test - add test case for rolling controlled shutdown
[ https://issues.apache.org/jira/browse/KAFKA-868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-868: Attachment: kafka-868-v1.patch Uploaded kafka-868-v1.patch > System Test - add test case for rolling controlled shutdown > --- > > Key: KAFKA-868 > URL: https://issues.apache.org/jira/browse/KAFKA-868 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Labels: kafka-0.8, replication-testing > Attachments: kafka-868-v1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-868) System Test - add test case for rolling controlled shutdown
[ https://issues.apache.org/jira/browse/KAFKA-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13638258#comment-13638258 ] John Fung commented on KAFKA-868: - Adding a case (testcase_9061) to test rolling controlled shutdown as follow : 1. Start Zookeeper 2. Start Broker cluster (7 machines) with Replication factor 2, 5 topics, 2 partitions 3. Settings in #2 can be configured at clean_shutdown_testsuite/testcase_9061/testcase_9061_properties.json 4. For each iteration : 5. Start ConsoleConsumer in the background 6. Start ProducerPerformance in the background to keep sending messages to all topics 7. For each broker : 8. Call kafka.admin.ShutdownBroker 9. Bounce the broker 10.Call kafka-preferred-replica-election.sh 11.Call kafka-list-topic.sh to find the number of under replicated partitions 12.Validate data matched for this iteration 13.Validate for "Failed to send" messages in producer 14.Clean up producer & consumer logs > System Test - add test case for rolling controlled shutdown > --- > > Key: KAFKA-868 > URL: https://issues.apache.org/jira/browse/KAFKA-868 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Labels: kafka-0.8, replication-testing > Attachments: kafka-868-v1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-868) System Test - add test case for rolling controlled shutdown
John Fung created KAFKA-868: --- Summary: System Test - add test case for rolling controlled shutdown Key: KAFKA-868 URL: https://issues.apache.org/jira/browse/KAFKA-868 Project: Kafka Issue Type: Task Reporter: John Fung -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-863) System Test - update 0.7 version of kafka-run-class.sh for Migration Tool test cases
[ https://issues.apache.org/jira/browse/KAFKA-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-863: Status: Patch Available (was: Open) Uploaded kafka-863-v1.patch > System Test - update 0.7 version of kafka-run-class.sh for Migration Tool > test cases > > > Key: KAFKA-863 > URL: https://issues.apache.org/jira/browse/KAFKA-863 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Labels: kafka-0.8, replication-testing > Attachments: kafka-863-v1.patch > > > The 0.7 version is located at: > system_test/migration_tool_testsuite/0.7/bin/kafka-run-class.sh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-863) System Test - update 0.7 version of kafka-run-class.sh for Migration Tool test cases
[ https://issues.apache.org/jira/browse/KAFKA-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-863: Attachment: kafka-863-v1.patch > System Test - update 0.7 version of kafka-run-class.sh for Migration Tool > test cases > > > Key: KAFKA-863 > URL: https://issues.apache.org/jira/browse/KAFKA-863 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Labels: kafka-0.8, replication-testing > Attachments: kafka-863-v1.patch > > > The 0.7 version is located at: > system_test/migration_tool_testsuite/0.7/bin/kafka-run-class.sh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-863) System Test - update 0.7 version of kafka-run-class.sh for Migration Tool test cases
[ https://issues.apache.org/jira/browse/KAFKA-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-863: Attachment: (was: kafka-run-class.sh) > System Test - update 0.7 version of kafka-run-class.sh for Migration Tool > test cases > > > Key: KAFKA-863 > URL: https://issues.apache.org/jira/browse/KAFKA-863 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Labels: kafka-0.8, replication-testing > > The 0.7 version is located at: > system_test/migration_tool_testsuite/0.7/bin/kafka-run-class.sh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-863) System Test - update 0.7 version of kafka-run-class.sh for Migration Tool test cases
[ https://issues.apache.org/jira/browse/KAFKA-863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-863: Attachment: kafka-run-class.sh > System Test - update 0.7 version of kafka-run-class.sh for Migration Tool > test cases > > > Key: KAFKA-863 > URL: https://issues.apache.org/jira/browse/KAFKA-863 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Labels: kafka-0.8, replication-testing > > The 0.7 version is located at: > system_test/migration_tool_testsuite/0.7/bin/kafka-run-class.sh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-863) System Test - update 0.7 version of kafka-run-class.sh for Migration Tool test cases
John Fung created KAFKA-863: --- Summary: System Test - update 0.7 version of kafka-run-class.sh for Migration Tool test cases Key: KAFKA-863 URL: https://issues.apache.org/jira/browse/KAFKA-863 Project: Kafka Issue Type: Task Reporter: John Fung The 0.7 version is located at: system_test/migration_tool_testsuite/0.7/bin/kafka-run-class.sh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-844: Attachment: kafka-844-v3.patch Thanks Neha for reviewing the patch. The Mirror Maker test cases are working fine with producer retries set to 3. Uploaded kafka-844-v3.patch to have message.send.max.retries=3 > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch, kafka-844-v2.patch, > kafka-844-v3.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-844: Attachment: kafka-844-v2.patch > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch, kafka-844-v2.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-844: Attachment: (was: kafka-844-v2.patch) > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620202#comment-13620202 ] John Fung edited comment on KAFKA-844 at 4/2/13 7:59 PM: - Thanks Neha for reviewing. The num.partitions is set to 5 and this patch is assumed to be applied on top of KAFKA-842. (original #3 (wait longer after mirror maker is terminated) is removed. was (Author: jfung): Thanks Neha for reviewing. The num.partitions is set to 5 and this patch is assumed to be applied on top of KAFKA-842. > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch, kafka-844-v2.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-844: Description: The following changes are needed to enhance System Test Mirror Maker cases: 1. Auto create topic 2. Specify num.partitions = 1 3. Add ConsoleConsumer to consume data from source cluster for validation reference was: The following changes are needed to enhance System Test Mirror Maker cases: 1. Auto create topic 2. Specify num.partitions = 1 3. Wait longer after mirror maker is terminated by kill -15 4. Add ConsoleConsumer to consume data from source cluster for validation reference > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch, kafka-844-v2.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620202#comment-13620202 ] John Fung commented on KAFKA-844: - Thanks Neha for reviewing. The num.partitions is set to 5 and this patch is assumed to be applied on top of KAFKA-842. > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch, kafka-844-v2.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Wait longer after mirror maker is terminated by kill -15 > 4. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-844: Attachment: kafka-844-v2.patch > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch, kafka-844-v2.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Wait longer after mirror maker is terminated by kill -15 > 4. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-844: Attachment: kafka-844-v1.patch > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Wait longer after mirror maker is terminated by kill -15 > 4. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-844) System Test - Mirror Maker cases enhancements
[ https://issues.apache.org/jira/browse/KAFKA-844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-844: Status: Patch Available (was: Open) > System Test - Mirror Maker cases enhancements > - > > Key: KAFKA-844 > URL: https://issues.apache.org/jira/browse/KAFKA-844 > Project: Kafka > Issue Type: Task >Reporter: John Fung > Attachments: kafka-844-v1.patch > > > The following changes are needed to enhance System Test Mirror Maker cases: > 1. Auto create topic > 2. Specify num.partitions = 1 > 3. Wait longer after mirror maker is terminated by kill -15 > 4. Add ConsoleConsumer to consume data from source cluster for validation > reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-844) System Test - Mirror Maker cases enhancements
John Fung created KAFKA-844: --- Summary: System Test - Mirror Maker cases enhancements Key: KAFKA-844 URL: https://issues.apache.org/jira/browse/KAFKA-844 Project: Kafka Issue Type: Task Reporter: John Fung The following changes are needed to enhance System Test Mirror Maker cases: 1. Auto create topic 2. Specify num.partitions = 1 3. Wait longer after mirror maker is terminated by kill -15 4. Add ConsoleConsumer to consume data from source cluster for validation reference -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-819) System Test : Add validation of log segment index to offset
[ https://issues.apache.org/jira/browse/KAFKA-819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-819: Status: Patch Available (was: Open) > System Test : Add validation of log segment index to offset > --- > > Key: KAFKA-819 > URL: https://issues.apache.org/jira/browse/KAFKA-819 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: kafka-0.8, replication-testing > Attachments: kafka-819-v1.patch > > > This can be achieved by calling DumpLogSegments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-819) System Test : Add validation of log segment index to offset
[ https://issues.apache.org/jira/browse/KAFKA-819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615809#comment-13615809 ] John Fung commented on KAFKA-819: - This patch is tested with an older 0.8 branch (before kafka-804 checkin) to reproduce the mismatching index-offset: 2013-03-27 14:48:33,033 - DEBUG - executing command [ssh localhost /home/kafka/kafka_sanity_20130314_gen_invalid_index/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --file /home/kafka/kafka_sanity_20130314_gen_invalid_index/system_test/replication_testsuite/testcase_0206/logs/broker-1/kafka_server_1_logs/test_1-2/.index --verify-index-only 2>&1] (kafka_system_test_utils) 2013-03-27 14:48:34,038 - DEBUG - error found [Mismatches in :.index] (kafka_system_test_utils) 2013-03-27 14:48:34,039 - DEBUG - [ Index offset: 755, log offset: 750] (kafka_system_test_utils) 2013-03-27 14:48:34,039 - DEBUG - [ Index offset: 635, log offset: 630] (kafka_system_test_utils) 2013-03-27 14:48:34,039 - DEBUG - [ Index offset: 515, log offset: 510] (kafka_system_test_utils) 2013-03-27 14:48:34,039 - DEBUG - [ Index offset: 395, log offset: 390] (kafka_system_test_utils) 2013-03-27 14:48:34,039 - DEBUG - [ Index offset: 275, log offset: 270] (kafka_system_test_utils) 2013-03-27 14:48:34,039 - DEBUG - [ Index offset: 125, log offset: 120] (kafka_system_test_utils) > System Test : Add validation of log segment index to offset > --- > > Key: KAFKA-819 > URL: https://issues.apache.org/jira/browse/KAFKA-819 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: kafka-0.8, replication-testing > Attachments: kafka-819-v1.patch > > > This can be achieved by calling DumpLogSegments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-819) System Test : Add validation of log segment index to offset
[ https://issues.apache.org/jira/browse/KAFKA-819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-819: Attachment: kafka-819-v1.patch Uploaded kafka-819-v1.patch > System Test : Add validation of log segment index to offset > --- > > Key: KAFKA-819 > URL: https://issues.apache.org/jira/browse/KAFKA-819 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: kafka-0.8, replication-testing > Attachments: kafka-819-v1.patch > > > This can be achieved by calling DumpLogSegments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-791) Fix validation bugs in System Test
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-791: Attachment: kafka-791-v4.patch Uploaded kafka-791-v4.patch with the following changes: 1. Added system_test_utils.diff_list to compare if 2 lists are identical 2. validate_simple_consumer_data_matched_across_replicas will use diff_list to make sure all messages received are in the same order and identical 3. Removed function "validate_simple_consumer_data_matched" from kafka_system_test_utils.py for clean up. > Fix validation bugs in System Test > -- > > Key: KAFKA-791 > URL: https://issues.apache.org/jira/browse/KAFKA-791 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > Attachments: kafka-791-v1.patch, kafka-791-v2.patch, > kafka-791-v3.patch, kafka-791-v4.patch > > > The following issues are found in data / log checksum match in System Test: > 1. kafka_system_test_utils.validate_simple_consumer_data_matched > It reports PASSED even some log segments don't match > 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched > in local Hudson for some time) > It reports PASSED in the Ack=1 cases even data loss is greater than the > tolerance (1%). > 3. kafka_system_test_utils.validate_simple_consumer_data_matched > It gets a unique set of MessageID to validate. It should leave all MessageID > as is (no dedup needed) and the test case should fail if sorted MessageID > don't match across the replicas. > 4. There is a data loss tolerance of 1% in the test cases of Ack=1. Currently > 1% is too strict and seeing some random failures due to 2 ~ 3% of data loss. > It will be increased to 5% such that the System Test will get a more > consistent passing rate in those test cases. The following will be updated to > 5% tolerance in kafka_system_test_utils: > validate_data_matched > validate_simple_consumer_data_matched > validate_data_matched_in_multi_topics_from_single_consumer_producer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-819) System Test : Add validation of log segment index to offset
John Fung created KAFKA-819: --- Summary: System Test : Add validation of log segment index to offset Key: KAFKA-819 URL: https://issues.apache.org/jira/browse/KAFKA-819 Project: Kafka Issue Type: Task Reporter: John Fung Assignee: John Fung This can be achieved by calling DumpLogSegments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-791) Fix validation bugs in System Test
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13605337#comment-13605337 ] John Fung commented on KAFKA-791: - Uploaded kafka-791-v3.patch with additional changes. The following validation functions : * validate_data_matched * validate_simple_consumer_data_matched * validate_simple_consumer_data_matched_across_replicas * validate_data_matched_in_multi_topics_from_single_consumer_producer are modified with these common behaviors : * producer MessageID list is always converted to a set (deduped) * consumer MessageID list is left as is * data loss / mismatch are compared by removing consumer MessageID from producer MessageID set * any duplicates in consumer MessageID will be treated as failure * Ack=1 test case data loss failure threshold is set to 5% * ordering of MessageID is not validated For the function validate_simple_consumer_data_matched_across_replicas : * compare each list (no dedupe) of consumer MessageID associated with its topic-partition in each replica * any MessageID mismatch in a certain topic-partition between replicas is reported as failure > Fix validation bugs in System Test > -- > > Key: KAFKA-791 > URL: https://issues.apache.org/jira/browse/KAFKA-791 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > Attachments: kafka-791-v1.patch, kafka-791-v2.patch, > kafka-791-v3.patch > > > The following issues are found in data / log checksum match in System Test: > 1. kafka_system_test_utils.validate_simple_consumer_data_matched > It reports PASSED even some log segments don't match > 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched > in local Hudson for some time) > It reports PASSED in the Ack=1 cases even data loss is greater than the > tolerance (1%). > 3. kafka_system_test_utils.validate_simple_consumer_data_matched > It gets a unique set of MessageID to validate. It should leave all MessageID > as is (no dedup needed) and the test case should fail if sorted MessageID > don't match across the replicas. > 4. There is a data loss tolerance of 1% in the test cases of Ack=1. Currently > 1% is too strict and seeing some random failures due to 2 ~ 3% of data loss. > It will be increased to 5% such that the System Test will get a more > consistent passing rate in those test cases. The following will be updated to > 5% tolerance in kafka_system_test_utils: > validate_data_matched > validate_simple_consumer_data_matched > validate_data_matched_in_multi_topics_from_single_consumer_producer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-791) Fix validation bugs in System Test
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-791: Attachment: kafka-791-v3.patch > Fix validation bugs in System Test > -- > > Key: KAFKA-791 > URL: https://issues.apache.org/jira/browse/KAFKA-791 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > Attachments: kafka-791-v1.patch, kafka-791-v2.patch, > kafka-791-v3.patch > > > The following issues are found in data / log checksum match in System Test: > 1. kafka_system_test_utils.validate_simple_consumer_data_matched > It reports PASSED even some log segments don't match > 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched > in local Hudson for some time) > It reports PASSED in the Ack=1 cases even data loss is greater than the > tolerance (1%). > 3. kafka_system_test_utils.validate_simple_consumer_data_matched > It gets a unique set of MessageID to validate. It should leave all MessageID > as is (no dedup needed) and the test case should fail if sorted MessageID > don't match across the replicas. > 4. There is a data loss tolerance of 1% in the test cases of Ack=1. Currently > 1% is too strict and seeing some random failures due to 2 ~ 3% of data loss. > It will be increased to 5% such that the System Test will get a more > consistent passing rate in those test cases. The following will be updated to > 5% tolerance in kafka_system_test_utils: > validate_data_matched > validate_simple_consumer_data_matched > validate_data_matched_in_multi_topics_from_single_consumer_producer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-791) Fix validation bugs in System Test
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-791: Attachment: kafka-791-v2.patch > Fix validation bugs in System Test > -- > > Key: KAFKA-791 > URL: https://issues.apache.org/jira/browse/KAFKA-791 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > Attachments: kafka-791-v1.patch, kafka-791-v2.patch > > > The following issues are found in data / log checksum match in System Test: > 1. kafka_system_test_utils.validate_simple_consumer_data_matched > It reports PASSED even some log segments don't match > 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched > in local Hudson for some time) > It reports PASSED in the Ack=1 cases even data loss is greater than the > tolerance (1%). > 3. kafka_system_test_utils.validate_simple_consumer_data_matched > It gets a unique set of MessageID to validate. It should leave all MessageID > as is (no dedup needed) and the test case should fail if sorted MessageID > don't match across the replicas. > 4. There is a data loss tolerance of 1% in the test cases of Ack=1. Currently > 1% is too strict and seeing some random failures due to 2 ~ 3% of data loss. > It will be increased to 5% such that the System Test will get a more > consistent passing rate in those test cases. The following will be updated to > 5% tolerance in kafka_system_test_utils: > validate_data_matched > validate_simple_consumer_data_matched > validate_data_matched_in_multi_topics_from_single_consumer_producer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-791) Fix validation bugs in System Test
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13601600#comment-13601600 ] John Fung commented on KAFKA-791: - Attached kafka-791-v2.patch with additional changes: kafka_system_test_utils.validate_simple_consumer_data_matched_across_replicas supports data validation in these scenarios : * multiple consumer entities * multiple topics in a single topic string separated by comma > Fix validation bugs in System Test > -- > > Key: KAFKA-791 > URL: https://issues.apache.org/jira/browse/KAFKA-791 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > Attachments: kafka-791-v1.patch > > > The following issues are found in data / log checksum match in System Test: > 1. kafka_system_test_utils.validate_simple_consumer_data_matched > It reports PASSED even some log segments don't match > 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched > in local Hudson for some time) > It reports PASSED in the Ack=1 cases even data loss is greater than the > tolerance (1%). > 3. kafka_system_test_utils.validate_simple_consumer_data_matched > It gets a unique set of MessageID to validate. It should leave all MessageID > as is (no dedup needed) and the test case should fail if sorted MessageID > don't match across the replicas. > 4. There is a data loss tolerance of 1% in the test cases of Ack=1. Currently > 1% is too strict and seeing some random failures due to 2 ~ 3% of data loss. > It will be increased to 5% such that the System Test will get a more > consistent passing rate in those test cases. The following will be updated to > 5% tolerance in kafka_system_test_utils: > validate_data_matched > validate_simple_consumer_data_matched > validate_data_matched_in_multi_topics_from_single_consumer_producer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-791) Fix validation bugs in System Test
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-791: Attachment: (was: kafka-791-v2.patch) > Fix validation bugs in System Test > -- > > Key: KAFKA-791 > URL: https://issues.apache.org/jira/browse/KAFKA-791 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > Attachments: kafka-791-v1.patch > > > The following issues are found in data / log checksum match in System Test: > 1. kafka_system_test_utils.validate_simple_consumer_data_matched > It reports PASSED even some log segments don't match > 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched > in local Hudson for some time) > It reports PASSED in the Ack=1 cases even data loss is greater than the > tolerance (1%). > 3. kafka_system_test_utils.validate_simple_consumer_data_matched > It gets a unique set of MessageID to validate. It should leave all MessageID > as is (no dedup needed) and the test case should fail if sorted MessageID > don't match across the replicas. > 4. There is a data loss tolerance of 1% in the test cases of Ack=1. Currently > 1% is too strict and seeing some random failures due to 2 ~ 3% of data loss. > It will be increased to 5% such that the System Test will get a more > consistent passing rate in those test cases. The following will be updated to > 5% tolerance in kafka_system_test_utils: > validate_data_matched > validate_simple_consumer_data_matched > validate_data_matched_in_multi_topics_from_single_consumer_producer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-791) Fix validation bugs in System Test
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-791: Attachment: kafka-791-v2.patch > Fix validation bugs in System Test > -- > > Key: KAFKA-791 > URL: https://issues.apache.org/jira/browse/KAFKA-791 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > Attachments: kafka-791-v1.patch, kafka-791-v2.patch > > > The following issues are found in data / log checksum match in System Test: > 1. kafka_system_test_utils.validate_simple_consumer_data_matched > It reports PASSED even some log segments don't match > 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched > in local Hudson for some time) > It reports PASSED in the Ack=1 cases even data loss is greater than the > tolerance (1%). > 3. kafka_system_test_utils.validate_simple_consumer_data_matched > It gets a unique set of MessageID to validate. It should leave all MessageID > as is (no dedup needed) and the test case should fail if sorted MessageID > don't match across the replicas. > 4. There is a data loss tolerance of 1% in the test cases of Ack=1. Currently > 1% is too strict and seeing some random failures due to 2 ~ 3% of data loss. > It will be increased to 5% such that the System Test will get a more > consistent passing rate in those test cases. The following will be updated to > 5% tolerance in kafka_system_test_utils: > validate_data_matched > validate_simple_consumer_data_matched > validate_data_matched_in_multi_topics_from_single_consumer_producer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-791) Fix validation bugs in System Test
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-791: Description: The following issues are found in data / log checksum match in System Test: 1. kafka_system_test_utils.validate_simple_consumer_data_matched It reports PASSED even some log segments don't match 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched in local Hudson for some time) It reports PASSED in the Ack=1 cases even data loss is greater than the tolerance (1%). 3. kafka_system_test_utils.validate_simple_consumer_data_matched It gets a unique set of MessageID to validate. It should leave all MessageID as is (no dedup needed) and the test case should fail if sorted MessageID don't match across the replicas. 4. There is a data loss tolerance of 1% in the test cases of Ack=1. Currently 1% is too strict and seeing some random failures due to 2 ~ 3% of data loss. It will be increased to 5% such that the System Test will get a more consistent passing rate in those test cases. The following will be updated to 5% tolerance in kafka_system_test_utils: validate_data_matched validate_simple_consumer_data_matched validate_data_matched_in_multi_topics_from_single_consumer_producer was: The following issues are found in data / log checksum match in System Test: 1. kafka_system_test_utils.validate_simple_consumer_data_matched It reports PASSED even some log segments don't match 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched in local Hudson for some time) It reports PASSED in the Ack=1 cases even data loss is greater than the tolerance (1%). 3. kafka_system_test_utils.validate_simple_consumer_data_matched It gets a unique set of MessageID to validate. It should leave all MessageID as is and the test case should fail if duplicates are detected. 4. There is a data loss tolerance of 1% in the test cases of Ack=1. Currently 1% is too strict and seeing some random failures due to 2 ~ 3% of data loss. It will be increased to 5% such that the System Test will get a more consistent passing rate in those test cases. The following will be updated to 5% tolerance in kafka_system_test_utils: validate_data_matched validate_simple_consumer_data_matched validate_data_matched_in_multi_topics_from_single_consumer_producer > Fix validation bugs in System Test > -- > > Key: KAFKA-791 > URL: https://issues.apache.org/jira/browse/KAFKA-791 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > Attachments: kafka-791-v1.patch > > > The following issues are found in data / log checksum match in System Test: > 1. kafka_system_test_utils.validate_simple_consumer_data_matched > It reports PASSED even some log segments don't match > 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched > in local Hudson for some time) > It reports PASSED in the Ack=1 cases even data loss is greater than the > tolerance (1%). > 3. kafka_system_test_utils.validate_simple_consumer_data_matched > It gets a unique set of MessageID to validate. It should leave all MessageID > as is (no dedup needed) and the test case should fail if sorted MessageID > don't match across the replicas. > 4. There is a data loss tolerance of 1% in the test cases of Ack=1. Currently > 1% is too strict and seeing some random failures due to 2 ~ 3% of data loss. > It will be increased to 5% such that the System Test will get a more > consistent passing rate in those test cases. The following will be updated to > 5% tolerance in kafka_system_test_utils: > validate_data_matched > validate_simple_consumer_data_matched > validate_data_matched_in_multi_topics_from_single_consumer_producer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-792) Update multiple attributes in testcase_xxxx_properties.json
[ https://issues.apache.org/jira/browse/KAFKA-792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-792: Attachment: kafka-792-v1.patch Attached kafka-792-v1.patch > Update multiple attributes in testcase__properties.json > --- > > Key: KAFKA-792 > URL: https://issues.apache.org/jira/browse/KAFKA-792 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > Attachments: kafka-792-v1.patch > > > The following are some of the properties need to be updated in some > testcase__properties.json. These changes have been patched in local > Hudson for a while. Create this new JIRA to check in these changes. > log.segment.bytes > default.replication.factor > num.partitions -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-792) Update multiple attributes in testcase_xxxx_properties.json
[ https://issues.apache.org/jira/browse/KAFKA-792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-792: Status: Patch Available (was: Open) > Update multiple attributes in testcase__properties.json > --- > > Key: KAFKA-792 > URL: https://issues.apache.org/jira/browse/KAFKA-792 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > Attachments: kafka-792-v1.patch > > > The following are some of the properties need to be updated in some > testcase__properties.json. These changes have been patched in local > Hudson for a while. Create this new JIRA to check in these changes. > log.segment.bytes > default.replication.factor > num.partitions -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-791) Fix validation bugs in System Test
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-791: Status: Patch Available (was: Open) > Fix validation bugs in System Test > -- > > Key: KAFKA-791 > URL: https://issues.apache.org/jira/browse/KAFKA-791 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > Attachments: kafka-791-v1.patch > > > The following issues are found in data / log checksum match in System Test: > 1. kafka_system_test_utils.validate_simple_consumer_data_matched > It reports PASSED even some log segments don't match > 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched > in local Hudson for some time) > It reports PASSED in the Ack=1 cases even data loss is greater than the > tolerance (1%). > 3. kafka_system_test_utils.validate_simple_consumer_data_matched > It gets a unique set of MessageID to validate. It should leave all MessageID > as is and the test case should fail if duplicates are detected. > 4. There is a data loss tolerance of 1% in the test cases of Ack=1. Currently > 1% is too strict and seeing some random failures due to 2 ~ 3% of data loss. > It will be increased to 5% such that the System Test will get a more > consistent passing rate in those test cases. The following will be updated to > 5% tolerance in kafka_system_test_utils: > validate_data_matched > validate_simple_consumer_data_matched > validate_data_matched_in_multi_topics_from_single_consumer_producer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-791) Fix validation bugs in System Test
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-791: Attachment: kafka-791-v1.patch Attached kafka-791-v1.patch > Fix validation bugs in System Test > -- > > Key: KAFKA-791 > URL: https://issues.apache.org/jira/browse/KAFKA-791 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > Attachments: kafka-791-v1.patch > > > The following issues are found in data / log checksum match in System Test: > 1. kafka_system_test_utils.validate_simple_consumer_data_matched > It reports PASSED even some log segments don't match > 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched > in local Hudson for some time) > It reports PASSED in the Ack=1 cases even data loss is greater than the > tolerance (1%). > 3. kafka_system_test_utils.validate_simple_consumer_data_matched > It gets a unique set of MessageID to validate. It should leave all MessageID > as is and the test case should fail if duplicates are detected. > 4. There is a data loss tolerance of 1% in the test cases of Ack=1. Currently > 1% is too strict and seeing some random failures due to 2 ~ 3% of data loss. > It will be increased to 5% such that the System Test will get a more > consistent passing rate in those test cases. The following will be updated to > 5% tolerance in kafka_system_test_utils: > validate_data_matched > validate_simple_consumer_data_matched > validate_data_matched_in_multi_topics_from_single_consumer_producer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-791) Fix validation bugs in System Test
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-791: Description: The following issues are found in data / log checksum match in System Test: 1. kafka_system_test_utils.validate_simple_consumer_data_matched It reports PASSED even some log segments don't match 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched in local Hudson for some time) It reports PASSED in the Ack=1 cases even data loss is greater than the tolerance (1%). 3. kafka_system_test_utils.validate_simple_consumer_data_matched It gets a unique set of MessageID to validate. It should leave all MessageID as is and the test case should fail if duplicates are detected. 4. There is a data loss tolerance of 1% in the test cases of Ack=1. Currently 1% is too strict and seeing some random failures due to 2 ~ 3% of data loss. It will be increased to 5% such that the System Test will get a more consistent passing rate in those test cases. The following will be updated to 5% tolerance in kafka_system_test_utils: validate_data_matched validate_simple_consumer_data_matched validate_data_matched_in_multi_topics_from_single_consumer_producer was: The following issues are found in data / log checksum match in System Test: 1. kafka_system_test_utils.validate_simple_consumer_data_matched It reports PASSED even some log segments don't match 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched in local Hudson for some time) It reports PASSED in the Ack=1 cases even data loss is greater than the tolerance (1%). 3. kafka_system_test_utils.validate_simple_consumer_data_matched It gets a unique set of MessageID to validate. It should leave all MessageID as is and the test case should fail if duplicates are detected. > Fix validation bugs in System Test > -- > > Key: KAFKA-791 > URL: https://issues.apache.org/jira/browse/KAFKA-791 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > > The following issues are found in data / log checksum match in System Test: > 1. kafka_system_test_utils.validate_simple_consumer_data_matched > It reports PASSED even some log segments don't match > 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched > in local Hudson for some time) > It reports PASSED in the Ack=1 cases even data loss is greater than the > tolerance (1%). > 3. kafka_system_test_utils.validate_simple_consumer_data_matched > It gets a unique set of MessageID to validate. It should leave all MessageID > as is and the test case should fail if duplicates are detected. > 4. There is a data loss tolerance of 1% in the test cases of Ack=1. Currently > 1% is too strict and seeing some random failures due to 2 ~ 3% of data loss. > It will be increased to 5% such that the System Test will get a more > consistent passing rate in those test cases. The following will be updated to > 5% tolerance in kafka_system_test_utils: > validate_data_matched > validate_simple_consumer_data_matched > validate_data_matched_in_multi_topics_from_single_consumer_producer -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-791) Fix validation bugs in System Test
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-791: Description: The following issues are found in data / log checksum match in System Test: 1. kafka_system_test_utils.validate_simple_consumer_data_matched It reports PASSED even some log segments don't match 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched in local Hudson for some time) It reports PASSED in the Ack=1 cases even data loss is greater than the tolerance (1%). 3. kafka_system_test_utils.validate_simple_consumer_data_matched It gets a unique set of MessageID to validate. It should leave all MessageID as is and the test case should fail if duplicates are detected. was: The following issues are found in data / log checksum match in System Test: 1. kafka_system_test_utils.validate_simple_consumer_data_matched It reports PASSED even some log segments don't match 2. kafka_system_test_utils.validate_data_matched It reports PASSED in the Ack=1 cases even data loss is greater than the tolerance (1%). 3. kafka_system_test_utils.validate_simple_consumer_data_matched It gets a unique set of MessageID to validate. It should leave all MessageID as is and the test case should fail if duplicates are detected. > Fix validation bugs in System Test > -- > > Key: KAFKA-791 > URL: https://issues.apache.org/jira/browse/KAFKA-791 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > > The following issues are found in data / log checksum match in System Test: > 1. kafka_system_test_utils.validate_simple_consumer_data_matched > It reports PASSED even some log segments don't match > 2. kafka_system_test_utils.validate_data_matched (this is fixed and patched > in local Hudson for some time) > It reports PASSED in the Ack=1 cases even data loss is greater than the > tolerance (1%). > 3. kafka_system_test_utils.validate_simple_consumer_data_matched > It gets a unique set of MessageID to validate. It should leave all MessageID > as is and the test case should fail if duplicates are detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-792) Update multiple attributes in testcase_xxxx_properties.json
[ https://issues.apache.org/jira/browse/KAFKA-792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-792: Description: The following are some of the properties need to be updated in some testcase__properties.json. These changes have been patched in local Hudson for a while. Create this new JIRA to check in these changes. log.segment.bytes default.replication.factor num.partitions was: The following are some of the properties need to be updated in some testcase__properties.json: log.segment.bytes default.replication.factor num.partitions > Update multiple attributes in testcase__properties.json > --- > > Key: KAFKA-792 > URL: https://issues.apache.org/jira/browse/KAFKA-792 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > > The following are some of the properties need to be updated in some > testcase__properties.json. These changes have been patched in local > Hudson for a while. Create this new JIRA to check in these changes. > log.segment.bytes > default.replication.factor > num.partitions -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-791) Fix validation bugs in System Test
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-791: Issue Type: Task (was: Bug) > Fix validation bugs in System Test > -- > > Key: KAFKA-791 > URL: https://issues.apache.org/jira/browse/KAFKA-791 > Project: Kafka > Issue Type: Task >Reporter: John Fung >Assignee: John Fung > Labels: replication-testing > > The following issues are found in data / log checksum match in System Test: > 1. kafka_system_test_utils.validate_simple_consumer_data_matched > It reports PASSED even some log segments don't match > 2. kafka_system_test_utils.validate_data_matched > It reports PASSED in the Ack=1 cases even data loss is greater than the > tolerance (1%). > 3. kafka_system_test_utils.validate_simple_consumer_data_matched > It gets a unique set of MessageID to validate. It should leave all MessageID > as is and the test case should fail if duplicates are detected. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
