[jira] [Commented] (HIVE-5029) direct SQL perf optimization cannot be tested well (yet)
[ https://issues.apache.org/jira/browse/HIVE-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13737294#comment-13737294 ] Sergey Shelukhin commented on HIVE-5029: The test passes on my test machine on recent trunk > direct SQL perf optimization cannot be tested well (yet) > > > Key: HIVE-5029 > URL: https://issues.apache.org/jira/browse/HIVE-5029 > Project: Hive > Issue Type: Test >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Critical > Attachments: HIVE-5029.patch > > > HIVE-4051 introduced perf optimization that involves getting partitions > directly via SQL in metastore. Given that SQL queries might not work on all > datastores (and will not work on non-SQL ones), JDO fallback is in place. > Given that perf improvement is very large for short queries, it's on by > default. > However, there's a problem with tests with regard to that. If SQL code is > broken, tests may fall back to JDO and pass. If JDO code is broken, SQL might > allow tests to pass. > We are going to disable SQL by default before the testing problem is resolved. > There are several possible solultions: > 1) Separate build for this setting. Seems like an overkill... > 2) Enable by default; disable by default in tests, create a clone of > TestCliDriver with a subset of queries that will exercise the SQL path. > 3) Have some sort of test hook inside metastore that will run both ORM and > SQL and compare. > 3') Or make a subclass of ObjectStore that will do that. ObjectStore is > already pluggable. > 4) Write unit tests for one of the modes (JDO, as non-default?) and declare > that they are sufficient; disable fallback in tests. > 3' seems like the easiest. For now we will disable SQL by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5029) direct SQL perf optimization cannot be tested well (yet)
[ https://issues.apache.org/jira/browse/HIVE-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13737108#comment-13737108 ] Sergey Shelukhin commented on HIVE-5029: [~ashutoshc] Wdyt? I am running the test to see what's wrong now, could be one of the examples of working SQL masking non working JDO, this query was added fairly recently > direct SQL perf optimization cannot be tested well (yet) > > > Key: HIVE-5029 > URL: https://issues.apache.org/jira/browse/HIVE-5029 > Project: Hive > Issue Type: Test >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Critical > Attachments: HIVE-5029.patch > > > HIVE-4051 introduced perf optimization that involves getting partitions > directly via SQL in metastore. Given that SQL queries might not work on all > datastores (and will not work on non-SQL ones), JDO fallback is in place. > Given that perf improvement is very large for short queries, it's on by > default. > However, there's a problem with tests with regard to that. If SQL code is > broken, tests may fall back to JDO and pass. If JDO code is broken, SQL might > allow tests to pass. > We are going to disable SQL by default before the testing problem is resolved. > There are several possible solultions: > 1) Separate build for this setting. Seems like an overkill... > 2) Enable by default; disable by default in tests, create a clone of > TestCliDriver with a subset of queries that will exercise the SQL path. > 3) Have some sort of test hook inside metastore that will run both ORM and > SQL and compare. > 3') Or make a subclass of ObjectStore that will do that. ObjectStore is > already pluggable. > 4) Write unit tests for one of the modes (JDO, as non-default?) and declare > that they are sufficient; disable fallback in tests. > 3' seems like the easiest. For now we will disable SQL by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5029) direct SQL perf optimization cannot be tested well (yet)
[
https://issues.apache.org/jira/browse/HIVE-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13734462#comment-13734462
]
Hive QA commented on HIVE-5029:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12596952/HIVE-5029.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 2772 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/351/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/351/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
> direct SQL perf optimization cannot be tested well (yet)
>
>
> Key: HIVE-5029
> URL: https://issues.apache.org/jira/browse/HIVE-5029
> Project: Hive
> Issue Type: Test
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Critical
> Attachments: HIVE-5029.patch
>
>
> HIVE-4051 introduced perf optimization that involves getting partitions
> directly via SQL in metastore. Given that SQL queries might not work on all
> datastores (and will not work on non-SQL ones), JDO fallback is in place.
> Given that perf improvement is very large for short queries, it's on by
> default.
> However, there's a problem with tests with regard to that. If SQL code is
> broken, tests may fall back to JDO and pass. If JDO code is broken, SQL might
> allow tests to pass.
> We are going to disable SQL by default before the testing problem is resolved.
> There are several possible solultions:
> 1) Separate build for this setting. Seems like an overkill...
> 2) Enable by default; disable by default in tests, create a clone of
> TestCliDriver with a subset of queries that will exercise the SQL path.
> 3) Have some sort of test hook inside metastore that will run both ORM and
> SQL and compare.
> 3') Or make a subclass of ObjectStore that will do that. ObjectStore is
> already pluggable.
> 4) Write unit tests for one of the modes (JDO, as non-default?) and declare
> that they are sufficient; disable fallback in tests.
> 3' seems like the easiest. For now we will disable SQL by default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
