[GitHub] yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-435898739 @pnowojski updated (squashed commits for `pull --rebase`) and I think what you mean is `flink-streaming-java` and `flink-streaming-scala`, right? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test
cases for modern Kafka connectors
URL: https://github.com/apache/flink/pull/6924#issuecomment-434539461
@pnowojski I can give you more details :
added `set -x` and `set -e`:
```
+ stop_kafka_cluster
+
${FLINK_DIR}/flink-end-to-end-tests/test-scripts/temp-test-directory-01N/kafka_2.11-2.0.0/bin/kafka-server-stop.sh
+
${FLINK_DIR}/flink-end-to-end-tests/test-scripts/temp-test-directory-01N/kafka_2.11-2.0.0/bin/zookeeper-server-stop.sh
++ grep -i 'kafka\.Kafka'
++ jps -vl
++ grep java
++ grep -v grep
++ awk '{print $1}'
+ PIDS=55405
+ '[' '!' -z 55405 ']'
+ kill -s TERM 55405
++ jps -vl
++ grep java
++ grep -i QuorumPeerMain
++ grep -v grep
++ awk '{print $1}'
+ PIDS=
[FAIL] 'Modern Kafka end-to-end test' failed after 0 minutes and 57 seconds!
Test exited with exit code 1
Stopping taskexecutor daemon (pid: 56949) on host ${MY_HOST}.
Stopping standalonesession daemon (pid: 56518) on host ${MY_HOST}.
No zookeeper daemon to stop on host ${MY_HOST}.
```
But, just add `set -x`
```
+ stop_kafka_cluster
+
${FLINK_DIR}/flink-end-to-end-tests/test-scripts/temp-test-directory-21N/kafka_2.11-2.0.0/bin/kafka-server-stop.sh
+
${FLINK_DIR}/flink-end-to-end-tests/test-scripts/temp-test-directory-21N/kafka_2.11-2.0.0/bin/zookeeper-server-stop.sh
++ jps -vl
++ grep -i 'kafka\.Kafka'
++ grep java
++ grep -v grep
++ awk '{print $1}'
+ PIDS=62800
+ '[' '!' -z 62800 ']'
+ kill -s TERM 62800
++ jps -vl
++ grep java
++ grep -i QuorumPeerMain
++ grep -v grep
++ awk '{print $1}'
+ PIDS=
+ '[' '!' -z '' ']'
+ mv -f
${FLINK_DIR}/flink-dist/target/flink-1.7-SNAPSHOT-bin/flink-1.7-SNAPSHOT/conf/flink-conf.yaml.bak
${FLINK_DIR}/flink-dist/target/flink-1.7-SNAPSHOT-bin/flink-1.7-SNAPSHOT/conf/flink-conf.yaml
[PASS] 'Modern Kafka end-to-end test' passed after 1 minutes and 0 seconds!
Test exited with exit code 0.
Stopping taskexecutor daemon (pid: 64351) on host ${MY_HOST}.
Stopping standalonesession daemon (pid: 63917) on host ${MY_HOST}.
No zookeeper daemon to stop on host ${MY_HOST}.
Deleted all files under
${FLINK_DIR}/flink-dist/target/flink-1.7-SNAPSHOT-bin/flink-1.7-SNAPSHOT/log/
Deleted
${FLINK_DIR}/flink-end-to-end-tests/test-scripts/temp-test-directory-21N
[PASS] All tests passed
```
the problem (see comment in this function):
```shell
function stop_kafka_cluster {
$KAFKA_DIR/bin/kafka-server-stop.sh
$KAFKA_DIR/bin/zookeeper-server-stop.sh
PIDS=$(jps -vl | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk
'{print $1}')
if [ ! -z "$PIDS" ]; then
kill -s TERM $PIDS
fi
# here
PIDS=$(jps -vl | grep java | grep -i QuorumPeerMain | grep -v grep | awk
'{print $1}')
if [ ! -z "$PIDS" ]; then
kill -s TERM $PIDS
fi
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-434358753 @pnowojski I have debugged locally, and the new changes have fixed the shell error. Before fixing, even if I do not add `set -e`, it will report an error. Now, I add `set -e`, which still doesn't report any errors, but the result doesn't show 'pass'. When I remove the `set -e`, the result will show that the test is passed. Considering that there was no `set -e` in these scripts before, I think it was this sentence that caused the test to fail. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-434358753 @pnowojski I have debugged locally, and the new changes have fixed the shell error. Before fixing, even if I do not add `set -e`, it will report an error. Now, I add `set -e`, which still doesn't report any errors, but the result doesn't show 'pass'. When I remove the `set -e`, the result will show that the test is passed. Considering that there was no `set-e' in these scripts before, I think it was this sentence that caused the test to fail. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-433904230 Let me try to locate it. This code was written by @tzulitai before. It seems the call : ``` $(get_partition_end_offset test-input 1) ``` may get a `null` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-433887813 @pnowojski I have fixed some of the issues you mentioned. Regarding the shell error, I don't actually change the logic. I have seen it in my local tests, but not every time. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
yanghua edited a comment on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-433394209 @pnowojski Using maven-shade-plugin, it's hard to package a jar (in `flink-example-streaming`) that doesn't contain flink kafka 0.10 or old kafka client, which is why FLINK-10107 appears. Unless we adopt the maven profile. But this is not very suitable. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
