[jira] [Comment Edited] (FLINK-33443) Make the test "testWriteComplexType" stable
[
https://issues.apache.org/jira/browse/FLINK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17790153#comment-17790153
]
huiyang chi edited comment on FLINK-33443 at 11/27/23 4:03 PM:
---
Okay! have a good day! :D
was (Author: JIRAUSER303032):
Okay! have a good day
> Make the test "testWriteComplexType" stable
> ---
>
> Key: FLINK-33443
> URL: https://issues.apache.org/jira/browse/FLINK-33443
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing to make the following test stable:
> {code:java}
> org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
> *STEPS TO REPRODUCE THE ISSUE:*
> * The following command can be run to execute the test with the
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive
> edu.illinois:nondex-maven-plugin:2.1.1:nondex
> -Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType
> {code}
> * The following error will be encountered:
> {code:java}
> [ERROR] Failures:
> [ERROR] HiveRunnerITCase.testWriteComplexType:166
> expected: "[1,2,3]{1:"a",2:"b"} {"f1":3,"f2":"c"}"
> but was: "[1,2,3]{2:"b",1:"a"} {"f1":3,"f2":"c"}"
> [INFO]
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
> *ROOT CAUSE ANALYSIS:*
> The test is currently flaky because of the assumption that the order of
> elements received in the _result_ variable will be consistent. There are
> currently two versions of query output that can be stored in _result._
> # The actual order that is expected where the output of the map attribute is
> \{1: "a", 2: "b"}.
> # The other order is the one shown in the error extract above where the
> ordering of the map attribute from the table is \{2: "b", 1: "a"}.
> *POTENTIAL FIX:*
> * The fix that I can suggest/have ready to raise a PR for is introducing
> another assertion on the second variant of the query output.
> * By asserting on whether the contents in _result_ are in one of the two
> orders, we can ascertain that the expected attributes with their contents are
> received as expected should the order in which they are received, not matter.
> Please share your thoughts on this finding and let me know if any other
> potential fix is possible for this test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33443) Make the test "testWriteComplexType" stable
[
https://issues.apache.org/jira/browse/FLINK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17790153#comment-17790153
]
huiyang chi commented on FLINK-33443:
-
Okay! have a good day
> Make the test "testWriteComplexType" stable
> ---
>
> Key: FLINK-33443
> URL: https://issues.apache.org/jira/browse/FLINK-33443
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing to make the following test stable:
> {code:java}
> org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
> *STEPS TO REPRODUCE THE ISSUE:*
> * The following command can be run to execute the test with the
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive
> edu.illinois:nondex-maven-plugin:2.1.1:nondex
> -Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType
> {code}
> * The following error will be encountered:
> {code:java}
> [ERROR] Failures:
> [ERROR] HiveRunnerITCase.testWriteComplexType:166
> expected: "[1,2,3]{1:"a",2:"b"} {"f1":3,"f2":"c"}"
> but was: "[1,2,3]{2:"b",1:"a"} {"f1":3,"f2":"c"}"
> [INFO]
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
> *ROOT CAUSE ANALYSIS:*
> The test is currently flaky because of the assumption that the order of
> elements received in the _result_ variable will be consistent. There are
> currently two versions of query output that can be stored in _result._
> # The actual order that is expected where the output of the map attribute is
> \{1: "a", 2: "b"}.
> # The other order is the one shown in the error extract above where the
> ordering of the map attribute from the table is \{2: "b", 1: "a"}.
> *POTENTIAL FIX:*
> * The fix that I can suggest/have ready to raise a PR for is introducing
> another assertion on the second variant of the query output.
> * By asserting on whether the contents in _result_ are in one of the two
> orders, we can ascertain that the expected attributes with their contents are
> received as expected should the order in which they are received, not matter.
> Please share your thoughts on this finding and let me know if any other
> potential fix is possible for this test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33443) Make the test "testWriteComplexType" stable
[
https://issues.apache.org/jira/browse/FLINK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huiyang chi updated FLINK-33443:
Attachment: (was: image-2023-11-27-09-36-03-081.png)
> Make the test "testWriteComplexType" stable
> ---
>
> Key: FLINK-33443
> URL: https://issues.apache.org/jira/browse/FLINK-33443
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing to make the following test stable:
> {code:java}
> org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
> *STEPS TO REPRODUCE THE ISSUE:*
> * The following command can be run to execute the test with the
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive
> edu.illinois:nondex-maven-plugin:2.1.1:nondex
> -Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType
> {code}
> * The following error will be encountered:
> {code:java}
> [ERROR] Failures:
> [ERROR] HiveRunnerITCase.testWriteComplexType:166
> expected: "[1,2,3]{1:"a",2:"b"} {"f1":3,"f2":"c"}"
> but was: "[1,2,3]{2:"b",1:"a"} {"f1":3,"f2":"c"}"
> [INFO]
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
> *ROOT CAUSE ANALYSIS:*
> The test is currently flaky because of the assumption that the order of
> elements received in the _result_ variable will be consistent. There are
> currently two versions of query output that can be stored in _result._
> # The actual order that is expected where the output of the map attribute is
> \{1: "a", 2: "b"}.
> # The other order is the one shown in the error extract above where the
> ordering of the map attribute from the table is \{2: "b", 1: "a"}.
> *POTENTIAL FIX:*
> * The fix that I can suggest/have ready to raise a PR for is introducing
> another assertion on the second variant of the query output.
> * By asserting on whether the contents in _result_ are in one of the two
> orders, we can ascertain that the expected attributes with their contents are
> received as expected should the order in which they are received, not matter.
> Please share your thoughts on this finding and let me know if any other
> potential fix is possible for this test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] (FLINK-33443) Make the test "testWriteComplexType" stable
[ https://issues.apache.org/jira/browse/FLINK-33443 ]
huiyang chi deleted comment on FLINK-33443:
-
was (Author: JIRAUSER303032):
Oh as this example:
!image-2023-11-27-09-36-03-081.png!
The direct producer might be the just simple change of the Map or the
environments (different JDK have different implementation). And except that
there are other kinds of reason we can detect too!
https://www.cs.cornell.edu/~legunsen/pubs/GyoriETAL16NonDexToolDemo.pdf
> Make the test "testWriteComplexType" stable
> ---
>
> Key: FLINK-33443
> URL: https://issues.apache.org/jira/browse/FLINK-33443
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
> Attachments: image-2023-11-27-09-36-03-081.png
>
>
> We are proposing to make the following test stable:
> {code:java}
> org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
> *STEPS TO REPRODUCE THE ISSUE:*
> * The following command can be run to execute the test with the
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive
> edu.illinois:nondex-maven-plugin:2.1.1:nondex
> -Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType
> {code}
> * The following error will be encountered:
> {code:java}
> [ERROR] Failures:
> [ERROR] HiveRunnerITCase.testWriteComplexType:166
> expected: "[1,2,3]{1:"a",2:"b"} {"f1":3,"f2":"c"}"
> but was: "[1,2,3]{2:"b",1:"a"} {"f1":3,"f2":"c"}"
> [INFO]
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
> *ROOT CAUSE ANALYSIS:*
> The test is currently flaky because of the assumption that the order of
> elements received in the _result_ variable will be consistent. There are
> currently two versions of query output that can be stored in _result._
> # The actual order that is expected where the output of the map attribute is
> \{1: "a", 2: "b"}.
> # The other order is the one shown in the error extract above where the
> ordering of the map attribute from the table is \{2: "b", 1: "a"}.
> *POTENTIAL FIX:*
> * The fix that I can suggest/have ready to raise a PR for is introducing
> another assertion on the second variant of the query output.
> * By asserting on whether the contents in _result_ are in one of the two
> orders, we can ascertain that the expected attributes with their contents are
> received as expected should the order in which they are received, not matter.
> Please share your thoughts on this finding and let me know if any other
> potential fix is possible for this test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33443) Make the test "testWriteComplexType" stable
[
https://issues.apache.org/jira/browse/FLINK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17790136#comment-17790136
]
huiyang chi commented on FLINK-33443:
-
Oh as this example:
!image-2023-11-27-09-36-03-081.png!
The direct producer might be the just simple change of the Map or the
environments (different JDK have different implementation). And except that
there are other kinds of reason we can detect too!
https://www.cs.cornell.edu/~legunsen/pubs/GyoriETAL16NonDexToolDemo.pdf
> Make the test "testWriteComplexType" stable
> ---
>
> Key: FLINK-33443
> URL: https://issues.apache.org/jira/browse/FLINK-33443
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
> Attachments: image-2023-11-27-09-36-03-081.png
>
>
> We are proposing to make the following test stable:
> {code:java}
> org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
> *STEPS TO REPRODUCE THE ISSUE:*
> * The following command can be run to execute the test with the
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive
> edu.illinois:nondex-maven-plugin:2.1.1:nondex
> -Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType
> {code}
> * The following error will be encountered:
> {code:java}
> [ERROR] Failures:
> [ERROR] HiveRunnerITCase.testWriteComplexType:166
> expected: "[1,2,3]{1:"a",2:"b"} {"f1":3,"f2":"c"}"
> but was: "[1,2,3]{2:"b",1:"a"} {"f1":3,"f2":"c"}"
> [INFO]
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
> *ROOT CAUSE ANALYSIS:*
> The test is currently flaky because of the assumption that the order of
> elements received in the _result_ variable will be consistent. There are
> currently two versions of query output that can be stored in _result._
> # The actual order that is expected where the output of the map attribute is
> \{1: "a", 2: "b"}.
> # The other order is the one shown in the error extract above where the
> ordering of the map attribute from the table is \{2: "b", 1: "a"}.
> *POTENTIAL FIX:*
> * The fix that I can suggest/have ready to raise a PR for is introducing
> another assertion on the second variant of the query output.
> * By asserting on whether the contents in _result_ are in one of the two
> orders, we can ascertain that the expected attributes with their contents are
> received as expected should the order in which they are received, not matter.
> Please share your thoughts on this finding and let me know if any other
> potential fix is possible for this test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33443) Make the test "testWriteComplexType" stable
[
https://issues.apache.org/jira/browse/FLINK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huiyang chi updated FLINK-33443:
Attachment: image-2023-11-27-09-36-03-081.png
> Make the test "testWriteComplexType" stable
> ---
>
> Key: FLINK-33443
> URL: https://issues.apache.org/jira/browse/FLINK-33443
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
> Attachments: image-2023-11-27-09-36-03-081.png
>
>
> We are proposing to make the following test stable:
> {code:java}
> org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
> *STEPS TO REPRODUCE THE ISSUE:*
> * The following command can be run to execute the test with the
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive
> edu.illinois:nondex-maven-plugin:2.1.1:nondex
> -Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType
> {code}
> * The following error will be encountered:
> {code:java}
> [ERROR] Failures:
> [ERROR] HiveRunnerITCase.testWriteComplexType:166
> expected: "[1,2,3]{1:"a",2:"b"} {"f1":3,"f2":"c"}"
> but was: "[1,2,3]{2:"b",1:"a"} {"f1":3,"f2":"c"}"
> [INFO]
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
> *ROOT CAUSE ANALYSIS:*
> The test is currently flaky because of the assumption that the order of
> elements received in the _result_ variable will be consistent. There are
> currently two versions of query output that can be stored in _result._
> # The actual order that is expected where the output of the map attribute is
> \{1: "a", 2: "b"}.
> # The other order is the one shown in the error extract above where the
> ordering of the map attribute from the table is \{2: "b", 1: "a"}.
> *POTENTIAL FIX:*
> * The fix that I can suggest/have ready to raise a PR for is introducing
> another assertion on the second variant of the query output.
> * By asserting on whether the contents in _result_ are in one of the two
> orders, we can ascertain that the expected attributes with their contents are
> received as expected should the order in which they are received, not matter.
> Please share your thoughts on this finding and let me know if any other
> potential fix is possible for this test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33443) Make the test "testWriteComplexType" stable
[
https://issues.apache.org/jira/browse/FLINK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789859#comment-17789859
]
huiyang chi edited comment on FLINK-33443 at 11/27/23 3:35 PM:
---
Hi Martijn : I can understand that currently this problem is not exposed in the
CI, because this kind of 80% problem we detected exposed mainly because the
change of the test environment (and more dangerous for lower-version JDK),
while for the CI the test environment won't change. In detail, ~50% of problem
we address is because the non-deterministic of the Hash (Set and Map), this is
one example Apache repo
[wicket|https://github.com/apache/wicket/commit/ed64e166dcba6715eafcbb7ca460d2b87e84cffc]
had encountered and addressed, this is kind of problem resolved for the
long-term gain :) . Here is a list of this kind of this problem :
https://github.com/TestingResearchIllinois/idoft
was (Author: JIRAUSER303032):
Hi Martijn : I can understand that currently this problem is not exposed in the
CI, because this kind of 80% problem we detected exposed mainly because the
change of the test environment (and more dangerous for lower-version JDK),
while for the CI the test environment won't change. In detail, ~50% of problem
we address is because the non-deterministic of the Hash (Set and Map), this is
one example Apache repo
[wicket|https://github.com/apache/wicket/commit/ed64e166dcba6715eafcbb7ca460d2b87e84cffc]
had encountered and addressed, this is kind of problem resolved for the
long-term gain :) . I also have some tests detected fixed:
https://github.com/MyEnthusiastic/flink/pulls
> Make the test "testWriteComplexType" stable
> ---
>
> Key: FLINK-33443
> URL: https://issues.apache.org/jira/browse/FLINK-33443
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing to make the following test stable:
> {code:java}
> org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
> *STEPS TO REPRODUCE THE ISSUE:*
> * The following command can be run to execute the test with the
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive
> edu.illinois:nondex-maven-plugin:2.1.1:nondex
> -Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType
> {code}
> * The following error will be encountered:
> {code:java}
> [ERROR] Failures:
> [ERROR] HiveRunnerITCase.testWriteComplexType:166
> expected: "[1,2,3]{1:"a",2:"b"} {"f1":3,"f2":"c"}"
> but was: "[1,2,3]{2:"b",1:"a"} {"f1":3,"f2":"c"}"
> [INFO]
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
> *ROOT CAUSE ANALYSIS:*
> The test is currently flaky because of the assumption that the order of
> elements received in the _result_ variable will be consistent. There are
> currently two versions of query output that can be stored in _result._
> # The actual order that is expected where the output of the map attribute is
> \{1: "a", 2: "b"}.
> # The other order is the one shown in the error extract above where the
> ordering of the map attribute from the table is \{2: "b", 1: "a"}.
> *POTENTIAL FIX:*
> * The fix that I can suggest/have ready to raise a PR for is introducing
> another assertion on the second variant of the query output.
> * By asserting on whether the contents in _result_ are in one of the two
> orders, we can ascertain that the expected attributes with their contents are
> received as expected should the order in which they are received, not matter.
> Please share your thoughts on this finding and let me know if any other
> potential fix is possible for this test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33443) Make the test "testWriteComplexType" stable
[
https://issues.apache.org/jira/browse/FLINK-33443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789859#comment-17789859
]
huiyang chi commented on FLINK-33443:
-
Hi Martijn : I can understand that currently this problem is not exposed in the
CI, because this kind of 80% problem we detected exposed mainly because the
change of the test environment (and more dangerous for lower-version JDK),
while for the CI the test environment won't change. In detail, ~50% of problem
we address is because the non-deterministic of the Hash (Set and Map), this is
one example Apache repo
[wicket|https://github.com/apache/wicket/commit/ed64e166dcba6715eafcbb7ca460d2b87e84cffc]
had encountered and addressed, this is kind of problem resolved for the
long-term gain :) . I also have some tests detected fixed:
https://github.com/MyEnthusiastic/flink/pulls
> Make the test "testWriteComplexType" stable
> ---
>
> Key: FLINK-33443
> URL: https://issues.apache.org/jira/browse/FLINK-33443
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Reporter: Krishna Anandan Ganesan
>Priority: Minor
>
> We are proposing to make the following test stable:
> {code:java}
> org.apache.flink.connectors.hive.HiveRunnerITCase.testWriteComplexType{code}
> *STEPS TO REPRODUCE THE ISSUE:*
> * The following command can be run to execute the test with the
> [NonDex|https://github.com/TestingResearchIllinois/NonDex] plugin:
> {code:java}
> mvn -pl flink-connectors/flink-connector-hive
> edu.illinois:nondex-maven-plugin:2.1.1:nondex
> -Dtest=org.apache.flink.connectors.hive.HiveRunnerITCase#testWriteComplexType
> {code}
> * The following error will be encountered:
> {code:java}
> [ERROR] Failures:
> [ERROR] HiveRunnerITCase.testWriteComplexType:166
> expected: "[1,2,3]{1:"a",2:"b"} {"f1":3,"f2":"c"}"
> but was: "[1,2,3]{2:"b",1:"a"} {"f1":3,"f2":"c"}"
> [INFO]
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0{code}
> *ROOT CAUSE ANALYSIS:*
> The test is currently flaky because of the assumption that the order of
> elements received in the _result_ variable will be consistent. There are
> currently two versions of query output that can be stored in _result._
> # The actual order that is expected where the output of the map attribute is
> \{1: "a", 2: "b"}.
> # The other order is the one shown in the error extract above where the
> ordering of the map attribute from the table is \{2: "b", 1: "a"}.
> *POTENTIAL FIX:*
> * The fix that I can suggest/have ready to raise a PR for is introducing
> another assertion on the second variant of the query output.
> * By asserting on whether the contents in _result_ are in one of the two
> orders, we can ascertain that the expected attributes with their contents are
> received as expected should the order in which they are received, not matter.
> Please share your thoughts on this finding and let me know if any other
> potential fix is possible for this test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
