[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[ https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380425#comment-16380425 ] Josh Elser commented on HIVE-18695: --- Thanks for the clarification, [~erwaman]. Would definitely be good to know if there is something wrong. Illl see if I can get my local environment set up again in the near future to poke at HIVE-18802. > fix TestAccumuloCliDriver.testCliDriver[accumulo_queries] > - > > Key: HIVE-18695 > URL: https://issues.apache.org/jira/browse/HIVE-18695 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > Attachments: HIVE-18695.01.patch > > > seems to be broken by HIVE-15680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[ https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379835#comment-16379835 ] Thejas M Nair commented on HIVE-18695: -- +1 > fix TestAccumuloCliDriver.testCliDriver[accumulo_queries] > - > > Key: HIVE-18695 > URL: https://issues.apache.org/jira/browse/HIVE-18695 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > Attachments: HIVE-18695.01.patch > > > seems to be broken by HIVE-15680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[ https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379649#comment-16379649 ] Anthony Hsu commented on HIVE-18695: +1 (non-binding – I am not a Hive committer) on your patch, [~kgyrtkirk] > fix TestAccumuloCliDriver.testCliDriver[accumulo_queries] > - > > Key: HIVE-18695 > URL: https://issues.apache.org/jira/browse/HIVE-18695 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > Attachments: HIVE-18695.01.patch > > > seems to be broken by HIVE-15680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[
https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379647#comment-16379647
]
Anthony Hsu commented on HIVE-18695:
[~elserj]
{quote}bq.[~erwaman], I'm confused on the phrasing: does HIVE-18802 apply both
before and after HIVE-15680? I think you're saying that it was an existing
problem, but it would also fix the test failures that HIVE-15680 caused?
{quote}
Yes, the bug reported in HIVE-18802 happens with or without the changes made in
HIVE-15680. I take back what I said about fixing HIVE-18802 also fixing
HIVE-15680. I think they are related but slightly different bugs. With
HIVE-18802, I think there's something funky going on with the way
AccumuloStorageHandler is handling predicate pushdown.
> fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
> -
>
> Key: HIVE-18695
> URL: https://issues.apache.org/jira/browse/HIVE-18695
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18695.01.patch
>
>
> seems to be broken by HIVE-15680
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[
https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377819#comment-16377819
]
Hive QA commented on HIVE-18695:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12912123/HIVE-18695.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 31 failed/errored test(s), 13414 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[
https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377757#comment-16377757
]
Hive QA commented on HIVE-18695:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
36s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9375/dev-support/hive-personality.sh
|
| git revision | master / 99b2f3c |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9375/yetus/patch-asflicense-problems.txt
|
| modules | C: accumulo-handler U: accumulo-handler |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9375/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
> -
>
> Key: HIVE-18695
> URL: https://issues.apache.org/jira/browse/HIVE-18695
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18695.01.patch
>
>
> seems to be broken by HIVE-15680
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[ https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377596#comment-16377596 ] Zoltan Haindrich commented on HIVE-18695: - I've attached a patch which disables ppd in that test > fix TestAccumuloCliDriver.testCliDriver[accumulo_queries] > - > > Key: HIVE-18695 > URL: https://issues.apache.org/jira/browse/HIVE-18695 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > Attachments: HIVE-18695.01.patch > > > seems to be broken by HIVE-15680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[
https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377592#comment-16377592
]
Zoltan Haindrich commented on HIVE-18695:
-
I've tried the magic ppd disabling combo of
{code}
set hive.optimize.index.filter=false;
set hive.optimize.ppd=false;
{code}
and it also caused a result difference ; but it seems like it just fixed an
existing incorrect result already present in the golden file (12 may not be in
that resultset)...
https://github.com/apache/hive/blob/99b2f3c7b890fd68dab6ca315988a77ade435984/accumulo-handler/src/test/results/positive/accumulo_queries.q.out#L386
> fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
> -
>
> Key: HIVE-18695
> URL: https://issues.apache.org/jira/browse/HIVE-18695
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18695.01.patch
>
>
> seems to be broken by HIVE-15680
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[
https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377497#comment-16377497
]
Josh Elser commented on HIVE-18695:
---
bq. From the comments above, it seems like a AccumuloStorageHandler bug where
it seems to depend on PPD having been done ( I haven't done a deeper
investigation, which would be necessary to confirm).
If memory serves me, yes, the storage handler is expecting that PPD (which I'm
reading as predicate pushdown, but am honestly not positive that's what the
acronym is ;)) happens. Accumulo has a server-side computation layer in which
we take some operands from the Hive query and can push them into Accumulo for
efficiency's sake. Nothing more than filters is happening now (e.g. {{key ==
1}} and {{key == 2}} in the above query), again, if memory serves. Is there
some new kind of API which the storage handlers can look at to figure out when
PPD is going to happen (or not)? That would have serious correctness issues if
not..
[~erwaman], I'm confused on the phrasing: does HIVE-18802 apply both before and
after HIVE-15680? I think you're saying that it was an existing problem, but it
would also fix the test failures that HIVE-15680 caused?
> fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
> -
>
> Key: HIVE-18695
> URL: https://issues.apache.org/jira/browse/HIVE-18695
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Priority: Major
>
> seems to be broken by HIVE-15680
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[ https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377487#comment-16377487 ] Thejas M Nair commented on HIVE-18695: -- Reverting HIVE-15680 can cause correctness issues with PPD + MR. [~erwaman] Is this caused by underlying bug in PPD or AccumuloStorageHandler or elsewhere ? >From the comments above, it seems like a AccumuloStorageHandler bug where it >seems to depend on PPD having been done ( I haven't done a deeper >investigation, which would be necessary to confirm). I think we should disable TestAccumuloCliDriver.testCliDriver[accumulo_queries] and track the issue via HIVE-18802 cc [~elserj] > fix TestAccumuloCliDriver.testCliDriver[accumulo_queries] > - > > Key: HIVE-18695 > URL: https://issues.apache.org/jira/browse/HIVE-18695 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > > seems to be broken by HIVE-15680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[ https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377392#comment-16377392 ] Anthony Hsu commented on HIVE-18695: [~kgyrtkirk], I didn't see an easy way to fix this test without breaking HIVE-15680, so for now, I think the easiest solution is to just revert the entire HIVE-15680 patch. > fix TestAccumuloCliDriver.testCliDriver[accumulo_queries] > - > > Key: HIVE-18695 > URL: https://issues.apache.org/jira/browse/HIVE-18695 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > > seems to be broken by HIVE-15680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[ https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376618#comment-16376618 ] Zoltan Haindrich commented on HIVE-18695: - [~erwaman]: Thank you for looking into it! From your description I think that thing should be handled in a separate ticket... Can you add a patch to disable ppd or something for this accumulo test (to prevent incorrect results); and leave a note in the new ticket about that in HIVE-18802 ? > fix TestAccumuloCliDriver.testCliDriver[accumulo_queries] > - > > Key: HIVE-18695 > URL: https://issues.apache.org/jira/browse/HIVE-18695 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > > seems to be broken by HIVE-15680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[
https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376402#comment-16376402
]
Anthony Hsu commented on HIVE-18695:
Hi [~kgyrtkirk], I investigated this further and the change in HIVE-15680 that
broke accumulo_queries.q is
{noformat}
// disable filter pushdown for mapreduce when there are more than one table
aliases,
// since we don't clone jobConf per alias
if (mrwork != null && mrwork.getAliases() != null && mrwork.getAliases().size()
> 1 &&
jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
return;
}{noformat}
In the case of the Accumulo CliDriver test, the execution engine is set to
"mr", so the "return" here is triggered, and then the subsequent code that sets
the filter expressions on the TableScanDesc is not triggered.
However, though removing the above fixes the test, I found a more serious
problem that still remains. If the same Accumulo table is referenced multiple
times in the same query, you get very strange results. Here's an example:
{noformat}
DROP TABLE accumulo_test;
CREATE TABLE accumulo_test(key int, value int)
STORED BY 'org.apache.hadoop.hive.accumulo.AccumuloStorageHandler'
WITH SERDEPROPERTIES ("accumulo.columns.mapping" = ":rowID,cf:string")
TBLPROPERTIES ("accumulo.table.name" = "accumulo_table_0");
INSERT OVERWRITE TABLE accumulo_test VALUES (0,0), (1,1), (2,2), (3,3);
SELECT * from accumulo_test where key == 1 union all select * from
accumulo_test where key == 2;{noformat}
The expected output is
{code:java}
1 1
2 2{code}
but the actual output is
{code:java}
1 0
1 1
1 2
1 3
2 0
2 1
2 2
2 3
{code}
I'll file a separate ticket for this issue. I think a fix for this issue would
also fix HIVE-15680, but for now, you can revert HIVE-15680.
> fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
> -
>
> Key: HIVE-18695
> URL: https://issues.apache.org/jira/browse/HIVE-18695
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Priority: Major
>
> seems to be broken by HIVE-15680
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[ https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374514#comment-16374514 ] Zoltan Haindrich commented on HIVE-18695: - sure; but if the problem still persist on monday; I'm afraid I've to revert your patch ; we're already fully loaded with broken tests... > fix TestAccumuloCliDriver.testCliDriver[accumulo_queries] > - > > Key: HIVE-18695 > URL: https://issues.apache.org/jira/browse/HIVE-18695 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > > seems to be broken by HIVE-15680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[ https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373878#comment-16373878 ] Anthony Hsu commented on HIVE-18695: [~kgyrtkirk], thanks for sharing that link. I'll take a closer look this weekend and try to fix the problem. > fix TestAccumuloCliDriver.testCliDriver[accumulo_queries] > - > > Key: HIVE-18695 > URL: https://issues.apache.org/jira/browse/HIVE-18695 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > > seems to be broken by HIVE-15680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[ https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372492#comment-16372492 ] Zoltan Haindrich commented on HIVE-18695: - [~erwaman] don't be offended; but have *you* checked it? I've bisected this problem prior to opening this ticket - then you let a whole week just pass...without answering my last comment but sure...I can show it that you patch have broken that test: https://travis-ci.org/kgyrtkirk/hive/builds/344666807 believe me: I hate hunting down test failures not caused by myself... > fix TestAccumuloCliDriver.testCliDriver[accumulo_queries] > - > > Key: HIVE-18695 > URL: https://issues.apache.org/jira/browse/HIVE-18695 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > > seems to be broken by HIVE-15680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[ https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372284#comment-16372284 ] Anthony Hsu commented on HIVE-18695: [~kgyrtkirk], can you just run the tests after reverting HIVE-15680 to see if that fixes the tests? > fix TestAccumuloCliDriver.testCliDriver[accumulo_queries] > - > > Key: HIVE-18695 > URL: https://issues.apache.org/jira/browse/HIVE-18695 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > > seems to be broken by HIVE-15680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[ https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16371041#comment-16371041 ] Zoltan Haindrich commented on HIVE-18695: - [~erwaman] could you please check these result set differences caused by HIVE-15680? > fix TestAccumuloCliDriver.testCliDriver[accumulo_queries] > - > > Key: HIVE-18695 > URL: https://issues.apache.org/jira/browse/HIVE-18695 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > > seems to be broken by HIVE-15680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[ https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363769#comment-16363769 ] Zoltan Haindrich commented on HIVE-18695: - there are output differences; jenkins is unreachabel right now, but see any recent builds at https://builds.apache.org/job/PreCommit-HIVE-Build/ once its available > fix TestAccumuloCliDriver.testCliDriver[accumulo_queries] > - > > Key: HIVE-18695 > URL: https://issues.apache.org/jira/browse/HIVE-18695 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > > seems to be broken by HIVE-15680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[ https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363282#comment-16363282 ] Anthony Hsu commented on HIVE-18695: Hi [~kgyrtkirk], could you include an example stack trace of or a link to a test failure? > fix TestAccumuloCliDriver.testCliDriver[accumulo_queries] > - > > Key: HIVE-18695 > URL: https://issues.apache.org/jira/browse/HIVE-18695 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > > seems to be broken by HIVE-15680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18695) fix TestAccumuloCliDriver.testCliDriver[accumulo_queries]
[ https://issues.apache.org/jira/browse/HIVE-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362023#comment-16362023 ] Zoltan Haindrich commented on HIVE-18695: - fyi: [~erwaman] > fix TestAccumuloCliDriver.testCliDriver[accumulo_queries] > - > > Key: HIVE-18695 > URL: https://issues.apache.org/jira/browse/HIVE-18695 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Priority: Major > > seems to be broken by HIVE-15680 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
