[jira] [Commented] (IMPALA-8944) Update and re-enable S3PlannerTest

2019-09-19 Thread ASF subversion and git services (Jira)


[ 
https://issues.apache.org/jira/browse/IMPALA-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933614#comment-16933614
 ] 

ASF subversion and git services commented on IMPALA-8944:
-

Commit feed25084a999fe0a4e7b58b5264fce5829c43e7 in impala's branch 
refs/heads/master from stakiar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=feed250 ]

IMPALA-8944: Update and re-enable S3PlannerTest

Addresses several test infra issues that were preventing the
S3PlannerTest from running successfully. Disables a few tests that are
no longer working, and removes some planner checks that are no longer
applicable when running on S3. Specifically, this patch removes the
checks in PlannerTestBase#checkScanRangeLocations when running against
S3, because the planner no longer generates scan ranges; generation is
deferred to the scheduler (IMPALA-5931).

Replaces the old logic of specifying S3-specific fe/ tests with a
combination of JUnit Categories and Maven Profiles. The previous method
was broken and assumed that all S3-specific fe/ tests started with S3*.
The new approach removes that restriction and only requires S3-specific
JUnit tests to be tagged with the Java annotation
'@Category(S3Tests.class)' (entire classes or individual tests can be
tagged with the annotation).

Testing:
* Ran fe/ tests with TARGET_FILESYSTEM=s3

Change-Id: I1690b6c5346376cfd4845c72062cc237e0f9
Reviewed-on: http://gerrit.cloudera.org:8080/14248
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 


> Update and re-enable S3PlannerTest
> --
>
> Key: IMPALA-8944
> URL: https://issues.apache.org/jira/browse/IMPALA-8944
> Project: IMPALA
>  Issue Type: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> It looks like we don't run {{S3PlannerTest}} in our regular Jenkins jobs. 
> When run against a HDFS mini-cluster, they are skipped because the 
> {{TARGET_FILESYSTEM}} is not S3. On our S3 jobs, they don't run either 
> because we skip all fe/ tests (most of them don't work against S3 / assume 
> they are running on HDFS).
> A few things need to be fixed to get this working:
> * The test cases in {{S3PlannerTest}} need to be fixed
> * The Jenkins jobs that runs the S3 tests needs the ability to run specific 
> fe/ tests (e.g. just the {{S3PlannerTest}} and to skip the rest)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Commented] (IMPALA-8944) Update and re-enable S3PlannerTest

2019-09-17 Thread Sahil Takiar (Jira)


[ 
https://issues.apache.org/jira/browse/IMPALA-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931788#comment-16931788
 ] 

Sahil Takiar commented on IMPALA-8944:
--

I've got 14 of the original 17 unit tests in {{S3PlannerTest}} working. Instead 
of relying on a test filter in {{run-all-tests.sh}}, I decided to use JUnit 
Categories and Maven profiles to select the tests to run (achieves a similar 
affect as TestNG Groups). I think it is a more robust and straightforward way 
of running tests. Now any fe/ tests that should be run for S3 can simply be 
tagged with the Java annotation {{@Category(S3Tests.class)}}.

The failing {{S3PlannerTest}}-s are:
* org.apache.impala.planner.S3PlannerTest.testTpcds
* org.apache.impala.planner.S3PlannerTest.testTpch
* org.apache.impala.planner.S3PlannerTest.testJoinOrder
* org.apache.impala.planner.S3PlannerTest.testSubqueryRewrite

All are failing for non-trivial reasons - e.g. actual differences in the 
explain plans when running on S3 vs. HDFS data (e.g. differences in memory 
estimates, join orders, etc.). I've opened IMPALA-8949 to investigate this.

> Update and re-enable S3PlannerTest
> --
>
> Key: IMPALA-8944
> URL: https://issues.apache.org/jira/browse/IMPALA-8944
> Project: IMPALA
>  Issue Type: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> It looks like we don't run {{S3PlannerTest}} in our regular Jenkins jobs. 
> When run against a HDFS mini-cluster, they are skipped because the 
> {{TARGET_FILESYSTEM}} is not S3. On our S3 jobs, they don't run either 
> because we skip all fe/ tests (most of them don't work against S3 / assume 
> they are running on HDFS).
> A few things need to be fixed to get this working:
> * The test cases in {{S3PlannerTest}} need to be fixed
> * The Jenkins jobs that runs the S3 tests needs the ability to run specific 
> fe/ tests (e.g. just the {{S3PlannerTest}} and to skip the rest)



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Commented] (IMPALA-8944) Update and re-enable S3PlannerTest

2019-09-16 Thread Sahil Takiar (Jira)


[ 
https://issues.apache.org/jira/browse/IMPALA-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930782#comment-16930782
 ] 

Sahil Takiar commented on IMPALA-8944:
--

Getting {{S3PlannerTest#testS3ScanRanges}} is not easily do-able because of 
IMPALA-5931 (scan range generation is no longer done in the fe/, it was moved 
to the scheduler).

> Update and re-enable S3PlannerTest
> --
>
> Key: IMPALA-8944
> URL: https://issues.apache.org/jira/browse/IMPALA-8944
> Project: IMPALA
>  Issue Type: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> It looks like we don't run {{S3PlannerTest}} in our regular Jenkins jobs. 
> When run against a HDFS mini-cluster, they are skipped because the 
> {{TARGET_FILESYSTEM}} is not S3. On our S3 jobs, they don't run either 
> because we skip all fe/ tests (most of them don't work against S3 / assume 
> they are running on HDFS).
> A few things need to be fixed to get this working:
> * The test cases in {{S3PlannerTest}} need to be fixed
> * The Jenkins jobs that runs the S3 tests needs the ability to run specific 
> fe/ tests (e.g. just the {{S3PlannerTest}} and to skip the rest)



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org



[jira] [Commented] (IMPALA-8944) Update and re-enable S3PlannerTest

2019-09-13 Thread Sahil Takiar (Jira)


[ 
https://issues.apache.org/jira/browse/IMPALA-8944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929482#comment-16929482
 ] 

Sahil Takiar commented on IMPALA-8944:
--

Looks like there was an attempt to only run fe tests beginning with {{S3*}} the 
target filesystem is S3 - 
https://github.com/apache/impala/blob/master/bin/run-all-tests.sh#L198 - 
however it looks like the logic is broken by 
https://github.com/apache/impala/blob/master/bin/run-all-tests.sh#L205 - the 
second filter cancels the first out out, which is why this doesn't work. 

> Update and re-enable S3PlannerTest
> --
>
> Key: IMPALA-8944
> URL: https://issues.apache.org/jira/browse/IMPALA-8944
> Project: IMPALA
>  Issue Type: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> It looks like we don't run {{S3PlannerTest}} in our regular Jenkins jobs. 
> When run against a HDFS mini-cluster, they are skipped because the 
> {{TARGET_FILESYSTEM}} is not S3. On our S3 jobs, they don't run either 
> because we skip all fe/ tests (most of them don't work against S3 / assume 
> they are running on HDFS).
> A few things need to be fixed to get this working:
> * The test cases in {{S3PlannerTest}} need to be fixed
> * The Jenkins jobs that runs the S3 tests needs the ability to run specific 
> fe/ tests (e.g. just the {{S3PlannerTest}} and to skip the rest)



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org