[jira] [Commented] (IMPALA-8944) Update and re-enable S3PlannerTest
[ https://issues.apache.org/jira/browse/IMPALA-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933614#comment-16933614 ] ASF subversion and git services commented on IMPALA-8944: - Commit feed25084a999fe0a4e7b58b5264fce5829c43e7 in impala's branch refs/heads/master from stakiar [ https://gitbox.apache.org/repos/asf?p=impala.git;h=feed250 ] IMPALA-8944: Update and re-enable S3PlannerTest Addresses several test infra issues that were preventing the S3PlannerTest from running successfully. Disables a few tests that are no longer working, and removes some planner checks that are no longer applicable when running on S3. Specifically, this patch removes the checks in PlannerTestBase#checkScanRangeLocations when running against S3, because the planner no longer generates scan ranges; generation is deferred to the scheduler (IMPALA-5931). Replaces the old logic of specifying S3-specific fe/ tests with a combination of JUnit Categories and Maven Profiles. The previous method was broken and assumed that all S3-specific fe/ tests started with S3*. The new approach removes that restriction and only requires S3-specific JUnit tests to be tagged with the Java annotation '@Category(S3Tests.class)' (entire classes or individual tests can be tagged with the annotation). Testing: * Ran fe/ tests with TARGET_FILESYSTEM=s3 Change-Id: I1690b6c5346376cfd4845c72062cc237e0f9 Reviewed-on: http://gerrit.cloudera.org:8080/14248 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Update and re-enable S3PlannerTest > -- > > Key: IMPALA-8944 > URL: https://issues.apache.org/jira/browse/IMPALA-8944 > Project: IMPALA > Issue Type: Test >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > It looks like we don't run {{S3PlannerTest}} in our regular Jenkins jobs. > When run against a HDFS mini-cluster, they are skipped because the > {{TARGET_FILESYSTEM}} is not S3. On our S3 jobs, they don't run either > because we skip all fe/ tests (most of them don't work against S3 / assume > they are running on HDFS). > A few things need to be fixed to get this working: > * The test cases in {{S3PlannerTest}} need to be fixed > * The Jenkins jobs that runs the S3 tests needs the ability to run specific > fe/ tests (e.g. just the {{S3PlannerTest}} and to skip the rest) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8944) Update and re-enable S3PlannerTest
[ https://issues.apache.org/jira/browse/IMPALA-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931788#comment-16931788 ] Sahil Takiar commented on IMPALA-8944: -- I've got 14 of the original 17 unit tests in {{S3PlannerTest}} working. Instead of relying on a test filter in {{run-all-tests.sh}}, I decided to use JUnit Categories and Maven profiles to select the tests to run (achieves a similar affect as TestNG Groups). I think it is a more robust and straightforward way of running tests. Now any fe/ tests that should be run for S3 can simply be tagged with the Java annotation {{@Category(S3Tests.class)}}. The failing {{S3PlannerTest}}-s are: * org.apache.impala.planner.S3PlannerTest.testTpcds * org.apache.impala.planner.S3PlannerTest.testTpch * org.apache.impala.planner.S3PlannerTest.testJoinOrder * org.apache.impala.planner.S3PlannerTest.testSubqueryRewrite All are failing for non-trivial reasons - e.g. actual differences in the explain plans when running on S3 vs. HDFS data (e.g. differences in memory estimates, join orders, etc.). I've opened IMPALA-8949 to investigate this. > Update and re-enable S3PlannerTest > -- > > Key: IMPALA-8944 > URL: https://issues.apache.org/jira/browse/IMPALA-8944 > Project: IMPALA > Issue Type: Test >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > It looks like we don't run {{S3PlannerTest}} in our regular Jenkins jobs. > When run against a HDFS mini-cluster, they are skipped because the > {{TARGET_FILESYSTEM}} is not S3. On our S3 jobs, they don't run either > because we skip all fe/ tests (most of them don't work against S3 / assume > they are running on HDFS). > A few things need to be fixed to get this working: > * The test cases in {{S3PlannerTest}} need to be fixed > * The Jenkins jobs that runs the S3 tests needs the ability to run specific > fe/ tests (e.g. just the {{S3PlannerTest}} and to skip the rest) -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8944) Update and re-enable S3PlannerTest
[ https://issues.apache.org/jira/browse/IMPALA-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930782#comment-16930782 ] Sahil Takiar commented on IMPALA-8944: -- Getting {{S3PlannerTest#testS3ScanRanges}} is not easily do-able because of IMPALA-5931 (scan range generation is no longer done in the fe/, it was moved to the scheduler). > Update and re-enable S3PlannerTest > -- > > Key: IMPALA-8944 > URL: https://issues.apache.org/jira/browse/IMPALA-8944 > Project: IMPALA > Issue Type: Test >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > It looks like we don't run {{S3PlannerTest}} in our regular Jenkins jobs. > When run against a HDFS mini-cluster, they are skipped because the > {{TARGET_FILESYSTEM}} is not S3. On our S3 jobs, they don't run either > because we skip all fe/ tests (most of them don't work against S3 / assume > they are running on HDFS). > A few things need to be fixed to get this working: > * The test cases in {{S3PlannerTest}} need to be fixed > * The Jenkins jobs that runs the S3 tests needs the ability to run specific > fe/ tests (e.g. just the {{S3PlannerTest}} and to skip the rest) -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8944) Update and re-enable S3PlannerTest
[ https://issues.apache.org/jira/browse/IMPALA-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929482#comment-16929482 ] Sahil Takiar commented on IMPALA-8944: -- Looks like there was an attempt to only run fe tests beginning with {{S3*}} the target filesystem is S3 - https://github.com/apache/impala/blob/master/bin/run-all-tests.sh#L198 - however it looks like the logic is broken by https://github.com/apache/impala/blob/master/bin/run-all-tests.sh#L205 - the second filter cancels the first out out, which is why this doesn't work. > Update and re-enable S3PlannerTest > -- > > Key: IMPALA-8944 > URL: https://issues.apache.org/jira/browse/IMPALA-8944 > Project: IMPALA > Issue Type: Test >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > It looks like we don't run {{S3PlannerTest}} in our regular Jenkins jobs. > When run against a HDFS mini-cluster, they are skipped because the > {{TARGET_FILESYSTEM}} is not S3. On our S3 jobs, they don't run either > because we skip all fe/ tests (most of them don't work against S3 / assume > they are running on HDFS). > A few things need to be fixed to get this working: > * The test cases in {{S3PlannerTest}} need to be fixed > * The Jenkins jobs that runs the S3 tests needs the ability to run specific > fe/ tests (e.g. just the {{S3PlannerTest}} and to skip the rest) -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org