[jira] [Commented] (HIVE-15550) fix arglist logging in schematool
[ https://issues.apache.org/jira/browse/HIVE-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825624#comment-15825624 ] Lefty Leverenz commented on HIVE-15550: --- [~thejas], the commit to master omitted the JIRA number. Please update errata.txt for commit ea87e0f26112f0cdbe2fa06e3090fb7b49d81569. Thanks. Example of updating errata.txt: HIVE-11876. > fix arglist logging in schematool > - > > Key: HIVE-15550 > URL: https://issues.apache.org/jira/browse/HIVE-15550 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.1.1 >Reporter: anishek >Assignee: anishek >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-15550.1.patch > > > In DEBUG mode schemaTool prints the password to log file. > This is also seen if the user includes --verbose option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14827) Micro benchmark for Parquet vectorized reader
[ https://issues.apache.org/jira/browse/HIVE-14827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825626#comment-15825626 ] Colin Ma commented on HIVE-14827: - hi, [~stakiar], [~Ferd], I develop an initial patch for this micro benchmark. If you don't work on this ticket, please assign to me, and I'll update the patch later, thanks. > Micro benchmark for Parquet vectorized reader > - > > Key: HIVE-14827 > URL: https://issues.apache.org/jira/browse/HIVE-14827 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Sahil Takiar > > We need a microbenchmark to evaluate the throughput and execution time for > Parquet vectorized reader. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values
[
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825731#comment-15825731
]
Elliot West commented on HIVE-15434:
Hello [~namit], [~amareshwari], [~hagleitn], [~ashutoshc]. Apologies for the
spam, but git suggests that you've all made significant contributions to Hive's
union feature in the past. Might one of you care to review this?
> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
> Issue Type: New Feature
> Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete.
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP
> BY clauses will fail, and Hive does not define syntax to extract the tag or
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to
> Hive users should at least have the ability to inspect and extract values for
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}}
> field that matches the tag of the union will be populated with the value. In
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag
> number matches then the value is returned, if it does not, then null is
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15434) Add UDF to allow interrogation of uniontype values
[
https://issues.apache.org/jira/browse/HIVE-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825798#comment-15825798
]
Amareshwari Sriramadasu commented on HIVE-15434:
I had a look at the patch. Seems the new UDF is all new code and not touching
any of existing code. And there shouldn't be any issue checking this code in.
+1 from my side. UDF code also look fine.
[~hagleitn] or [~ashutoshc] Can you help have a look at new UDF if it has
taken care of practices in UDF writing?
> Add UDF to allow interrogation of uniontype values
> --
>
> Key: HIVE-15434
> URL: https://issues.apache.org/jira/browse/HIVE-15434
> Project: Hive
> Issue Type: New Feature
> Components: UDF
>Affects Versions: 2.1.1
>Reporter: David Maughan
>Assignee: David Maughan
> Attachments: HIVE-15434.01.patch, HIVE-15434.02.patch
>
>
> h2. Overview
> As stated in the documention:
> {quote}
> UNIONTYPE support is incomplete The UNIONTYPE datatype was introduced in Hive
> 0.7.0 (HIVE-537), but full support for this type in Hive remains incomplete.
> Queries that reference UNIONTYPE fields in JOIN (HIVE-2508), WHERE, and GROUP
> BY clauses will fail, and Hive does not define syntax to extract the tag or
> value fields of a UNIONTYPE. This means that UNIONTYPEs are effectively
> look-at-only.
> {quote}
> It is essential to have a usable uniontype. Until full support is added to
> Hive users should at least have the ability to inspect and extract values for
> further comparison or transformation.
> h2. Proposal
> I propose to add a GenericUDF that has 2 modes of operation. Consider the
> following schema and data that contains a union:
> Schema:
> {code}
> struct>
> {code}
> Query:
> {code}
> hive> select field1 from thing;
> {0:0}
> {1:"one"}
> {code}
> h4. Explode to Struct
> This method will recursively convert all unions within the type to structs
> with fields named {{tag_n}}, {{n}} being the tag number. Only the {{tag_*}}
> field that matches the tag of the union will be populated with the value. In
> the case above the schema of field1 will be converted to:
> {code}
> struct
> {code}
> {code}
> hive> select extract_union(field1) from thing;
> {"tag_0":0,"tag_1":null}
> {"tag_0":null,"tag_1":one}
> {code}
> {code}
> hive> select extract_union(field1).tag_0 from thing;
> 0
> null
> {code}
> h4. Extract the specified tag
> This method will simply extract the value of the specified tag. If the tag
> number matches then the value is returned, if it does not, then null is
> returned.
> {code}
> hive> select extract_union(field1, 0) from thing;
> 0
> null
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15478) Add file + checksum list for create table/partition during notification creation (whenever relevant)
[ https://issues.apache.org/jira/browse/HIVE-15478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-15478: -- Attachment: HIVE-15478.3.patch Fix UT and address RB comments. > Add file + checksum list for create table/partition during notification > creation (whenever relevant) > > > Key: HIVE-15478 > URL: https://issues.apache.org/jira/browse/HIVE-15478 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Daniel Dai > Attachments: HIVE-15478.1.patch, HIVE-15478.2.patch, > HIVE-15478.3.patch > > > Currently, file list is being generated during REPL DUMP which will result in > inconsistent data getting captured. This ticket is used for event dumping. > Bootstrap dump checksum will be in a different Jira. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15647) Combination of a boolean condition and null-safe comparison leads to NPE
[
https://issues.apache.org/jira/browse/HIVE-15647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825922#comment-15825922
]
Remus Rusanu commented on HIVE-15647:
-
Originally reported by Carter Shanklin
> Combination of a boolean condition and null-safe comparison leads to NPE
>
>
> Key: HIVE-15647
> URL: https://issues.apache.org/jira/browse/HIVE-15647
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>
> Here's a simple example with the foodmart database:
> {code}
> hive> explain select count(*) from
> > sales_fact_1997 join store on sales_fact_1997.store_id = store.store_id
> > where ((store.salad_bar)) and ((store_number) <=> (customer_id));
> FAILED: NullPointerException null
> {code}
> This happens on trunk and on HDP 2.5.3 / Hive 2. If you use = the NPE doesn't
> happen. If you remove the boolean condition the NPE doesn't happen.
> {code}
> FAILED: NullPointerException null
> 2016-12-13T18:23:33,604 ERROR [c4b7242e-1252-4709-8adf-22f631af75e8 main]
> ql.Driver: FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory$ConstantPropagateFilterProc.process(ConstantPropagateProcFactory.java:1047)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagate$ConstantPropagateWalker.walk(ConstantPropagate.java:151)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagate.transform(ConstantPropagate.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:242)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10913)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:246)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:75)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:435)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:326)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1169)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1095)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1083)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15647) Combination of a boolean condition and null-safe comparison leads to NPE
[
https://issues.apache.org/jira/browse/HIVE-15647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HIVE-15647:
Priority: Minor (was: Major)
> Combination of a boolean condition and null-safe comparison leads to NPE
>
>
> Key: HIVE-15647
> URL: https://issues.apache.org/jira/browse/HIVE-15647
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Remus Rusanu
>Assignee: Remus Rusanu
>Priority: Minor
>
> Here's a simple example with the foodmart database:
> {code}
> hive> explain select count(*) from
> > sales_fact_1997 join store on sales_fact_1997.store_id = store.store_id
> > where ((store.salad_bar)) and ((store_number) <=> (customer_id));
> FAILED: NullPointerException null
> {code}
> This happens on trunk and on HDP 2.5.3 / Hive 2. If you use = the NPE doesn't
> happen. If you remove the boolean condition the NPE doesn't happen.
> {code}
> FAILED: NullPointerException null
> 2016-12-13T18:23:33,604 ERROR [c4b7242e-1252-4709-8adf-22f631af75e8 main]
> ql.Driver: FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory$ConstantPropagateFilterProc.process(ConstantPropagateProcFactory.java:1047)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagate$ConstantPropagateWalker.walk(ConstantPropagate.java:151)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagate.transform(ConstantPropagate.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:242)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10913)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:246)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:75)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:435)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:326)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1169)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1095)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1083)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15647) Combination of a boolean condition and null-safe comparison leads to NPE
[
https://issues.apache.org/jira/browse/HIVE-15647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-15647:
---
Reporter: Carter Shanklin (was: Remus Rusanu)
> Combination of a boolean condition and null-safe comparison leads to NPE
>
>
> Key: HIVE-15647
> URL: https://issues.apache.org/jira/browse/HIVE-15647
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Carter Shanklin
>Assignee: Remus Rusanu
>Priority: Minor
>
> Here's a simple example with the foodmart database:
> {code}
> hive> explain select count(*) from
> > sales_fact_1997 join store on sales_fact_1997.store_id = store.store_id
> > where ((store.salad_bar)) and ((store_number) <=> (customer_id));
> FAILED: NullPointerException null
> {code}
> This happens on trunk and on HDP 2.5.3 / Hive 2. If you use = the NPE doesn't
> happen. If you remove the boolean condition the NPE doesn't happen.
> {code}
> FAILED: NullPointerException null
> 2016-12-13T18:23:33,604 ERROR [c4b7242e-1252-4709-8adf-22f631af75e8 main]
> ql.Driver: FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory$ConstantPropagateFilterProc.process(ConstantPropagateProcFactory.java:1047)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagate$ConstantPropagateWalker.walk(ConstantPropagate.java:151)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagate.transform(ConstantPropagate.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:242)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10913)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:246)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:75)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:435)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:326)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1169)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1095)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1083)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15648) Hive throws compilation error due to $powermock.version not being present in root pom
[ https://issues.apache.org/jira/browse/HIVE-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15648: Description: Looks like caused by HIVE-15550 > Hive throws compilation error due to $powermock.version not being present in > root pom > - > > Key: HIVE-15648 > URL: https://issues.apache.org/jira/browse/HIVE-15648 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > > Looks like caused by HIVE-15550 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15648) Hive throws compilation error due to $powermock.version not being present in root pom
[ https://issues.apache.org/jira/browse/HIVE-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15648: Attachment: HIVE-15648.1.patch > Hive throws compilation error due to $powermock.version not being present in > root pom > - > > Key: HIVE-15648 > URL: https://issues.apache.org/jira/browse/HIVE-15648 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-15648.1.patch > > > Looks like caused by HIVE-15550 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15648) Hive throws compilation error due to $powermock.version not being present in root pom
[ https://issues.apache.org/jira/browse/HIVE-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15648: Status: Patch Available (was: Open) > Hive throws compilation error due to $powermock.version not being present in > root pom > - > > Key: HIVE-15648 > URL: https://issues.apache.org/jira/browse/HIVE-15648 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-15648.1.patch > > > Looks like caused by HIVE-15550 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15648) Hive throws compilation error due to $powermock.version not being present in root pom
[ https://issues.apache.org/jira/browse/HIVE-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825963#comment-15825963 ] Vaibhav Gumashta commented on HIVE-15648: - [~thejas] [~anishek] can you please review? > Hive throws compilation error due to $powermock.version not being present in > root pom > - > > Key: HIVE-15648 > URL: https://issues.apache.org/jira/browse/HIVE-15648 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-15648.1.patch > > > Looks like caused by HIVE-15550 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15053) Beeline#addlocaldriver - reduce classpath scanning
[
https://issues.apache.org/jira/browse/HIVE-15053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825964#comment-15825964
]
Vaibhav Gumashta commented on HIVE-15053:
-
[~kgyrtkirk] Thanks a lot! Will take a look today.
> Beeline#addlocaldriver - reduce classpath scanning
> --
>
> Key: HIVE-15053
> URL: https://issues.apache.org/jira/browse/HIVE-15053
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Attachments: HIVE-15053.1.patch, HIVE-15053.1.patch,
> HIVE-15053.1.patch, HIVE-15053.2.patch
>
>
> There is a classpath scanning machinery inside {{ClassNameCompleter}}.
> I think the sole purpose of these things is to scan for jdbc drivers...(but
> not entirely sure)
> if it is indeed looking for jdbc drivers..then possibly this can be removed
> without any issues because modern jdbc drivers usually advertise their driver
> as a service-loadable class for {{java.sql.Driver}}
> http://www.onjava.com/2006/08/02/jjdbc-4-enhancements-in-java-se-6.html
> Auto-Loading of JDBC Driver
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15647) Combination of a boolean condition and null-safe comparison leads to NPE
[
https://issues.apache.org/jira/browse/HIVE-15647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HIVE-15647:
Status: Patch Available (was: Open)
> Combination of a boolean condition and null-safe comparison leads to NPE
>
>
> Key: HIVE-15647
> URL: https://issues.apache.org/jira/browse/HIVE-15647
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Carter Shanklin
>Assignee: Remus Rusanu
>Priority: Minor
> Attachments: HIVE-15647.01.patch
>
>
> Here's a simple example with the foodmart database:
> {code}
> hive> explain select count(*) from
> > sales_fact_1997 join store on sales_fact_1997.store_id = store.store_id
> > where ((store.salad_bar)) and ((store_number) <=> (customer_id));
> FAILED: NullPointerException null
> {code}
> This happens on trunk and on HDP 2.5.3 / Hive 2. If you use = the NPE doesn't
> happen. If you remove the boolean condition the NPE doesn't happen.
> {code}
> FAILED: NullPointerException null
> 2016-12-13T18:23:33,604 ERROR [c4b7242e-1252-4709-8adf-22f631af75e8 main]
> ql.Driver: FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory$ConstantPropagateFilterProc.process(ConstantPropagateProcFactory.java:1047)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagate$ConstantPropagateWalker.walk(ConstantPropagate.java:151)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagate.transform(ConstantPropagate.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:242)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10913)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:246)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:75)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:435)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:326)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1169)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1095)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1083)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15647) Combination of a boolean condition and null-safe comparison leads to NPE
[
https://issues.apache.org/jira/browse/HIVE-15647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Remus Rusanu updated HIVE-15647:
Attachment: HIVE-15647.01.patch
> Combination of a boolean condition and null-safe comparison leads to NPE
>
>
> Key: HIVE-15647
> URL: https://issues.apache.org/jira/browse/HIVE-15647
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Carter Shanklin
>Assignee: Remus Rusanu
>Priority: Minor
> Attachments: HIVE-15647.01.patch
>
>
> Here's a simple example with the foodmart database:
> {code}
> hive> explain select count(*) from
> > sales_fact_1997 join store on sales_fact_1997.store_id = store.store_id
> > where ((store.salad_bar)) and ((store_number) <=> (customer_id));
> FAILED: NullPointerException null
> {code}
> This happens on trunk and on HDP 2.5.3 / Hive 2. If you use = the NPE doesn't
> happen. If you remove the boolean condition the NPE doesn't happen.
> {code}
> FAILED: NullPointerException null
> 2016-12-13T18:23:33,604 ERROR [c4b7242e-1252-4709-8adf-22f631af75e8 main]
> ql.Driver: FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory$ConstantPropagateFilterProc.process(ConstantPropagateProcFactory.java:1047)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagate$ConstantPropagateWalker.walk(ConstantPropagate.java:151)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagate.transform(ConstantPropagate.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:242)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10913)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:246)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:75)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:435)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:326)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1169)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1095)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1083)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15647) Combination of a boolean condition and null-safe comparison leads to NPE
[
https://issues.apache.org/jira/browse/HIVE-15647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826100#comment-15826100
]
Remus Rusanu commented on HIVE-15647:
-
The issue seems to be originating from
ExprWalkerProcFactory::extractFinalCandidates():
{noformat}
ExprInfo exprInfo = ctx.getExprInfo(expr);
if (exprInfo != null && exprInfo.isCandidate) {
ctx.addFinalCandidate(exprInfo.alias, exprInfo.convertedExpr != null ?
exprInfo.convertedExpr : expr);
return;
}
{noformat}
When exprInfo.convertedExpr is used instead of expr, the original alias of expr
is kept. In the tested case this is null and this leads, after a convoluted
path, to the NPE (the expression is moved into non-candidates list as a
{null=[null]} map and this ultimately gets read as a descriptor).
> Combination of a boolean condition and null-safe comparison leads to NPE
>
>
> Key: HIVE-15647
> URL: https://issues.apache.org/jira/browse/HIVE-15647
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Carter Shanklin
>Assignee: Remus Rusanu
>Priority: Minor
> Attachments: HIVE-15647.01.patch
>
>
> Here's a simple example with the foodmart database:
> {code}
> hive> explain select count(*) from
> > sales_fact_1997 join store on sales_fact_1997.store_id = store.store_id
> > where ((store.salad_bar)) and ((store_number) <=> (customer_id));
> FAILED: NullPointerException null
> {code}
> This happens on trunk and on HDP 2.5.3 / Hive 2. If you use = the NPE doesn't
> happen. If you remove the boolean condition the NPE doesn't happen.
> {code}
> FAILED: NullPointerException null
> 2016-12-13T18:23:33,604 ERROR [c4b7242e-1252-4709-8adf-22f631af75e8 main]
> ql.Driver: FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory$ConstantPropagateFilterProc.process(ConstantPropagateProcFactory.java:1047)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagate$ConstantPropagateWalker.walk(ConstantPropagate.java:151)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagate.transform(ConstantPropagate.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:242)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10913)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:246)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:75)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:435)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:326)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1169)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1095)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1083)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15166) Provide beeline option to set the jline history max size
[ https://issues.apache.org/jira/browse/HIVE-15166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826111#comment-15826111 ] Aihua Xu commented on HIVE-15166: - The change looks good. Of course, I left a comment that jLine FileHistory constructor needs an improvement to take the max_size, since otherwise, it will load 500 lines and then resize to the specified number of lines. The load could potentially cause OOM, but with current jLine, it's hard to change that. +1 pending test. > Provide beeline option to set the jline history max size > > > Key: HIVE-15166 > URL: https://issues.apache.org/jira/browse/HIVE-15166 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.1.0 >Reporter: Eric Lin >Assignee: Eric Lin >Priority: Minor > Attachments: HIVE-15166.2.patch, HIVE-15166.patch > > > Currently Beeline does not provide an option to limit the max size for > beeline history file, in the case that each query is very big, it will flood > the history file and slow down beeline on start up and shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15647) Combination of a boolean condition and null-safe comparison leads to NPE
[
https://issues.apache.org/jira/browse/HIVE-15647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826116#comment-15826116
]
Remus Rusanu commented on HIVE-15647:
-
CR https://reviews.apache.org/r/55620/
> Combination of a boolean condition and null-safe comparison leads to NPE
>
>
> Key: HIVE-15647
> URL: https://issues.apache.org/jira/browse/HIVE-15647
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Carter Shanklin
>Assignee: Remus Rusanu
>Priority: Minor
> Attachments: HIVE-15647.01.patch
>
>
> Here's a simple example with the foodmart database:
> {code}
> hive> explain select count(*) from
> > sales_fact_1997 join store on sales_fact_1997.store_id = store.store_id
> > where ((store.salad_bar)) and ((store_number) <=> (customer_id));
> FAILED: NullPointerException null
> {code}
> This happens on trunk and on HDP 2.5.3 / Hive 2. If you use = the NPE doesn't
> happen. If you remove the boolean condition the NPE doesn't happen.
> {code}
> FAILED: NullPointerException null
> 2016-12-13T18:23:33,604 ERROR [c4b7242e-1252-4709-8adf-22f631af75e8 main]
> ql.Driver: FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagateProcFactory$ConstantPropagateFilterProc.process(ConstantPropagateProcFactory.java:1047)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagate$ConstantPropagateWalker.walk(ConstantPropagate.java:151)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.ConstantPropagate.transform(ConstantPropagate.java:120)
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:242)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10913)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:246)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:75)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:250)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:435)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:326)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1169)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1262)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1095)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1083)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15617) Improve the avg performance for Range based window
[ https://issues.apache.org/jira/browse/HIVE-15617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826178#comment-15826178 ] Aihua Xu commented on HIVE-15617: - Code review: https://reviews.apache.org/r/55623/ > Improve the avg performance for Range based window > -- > > Key: HIVE-15617 > URL: https://issues.apache.org/jira/browse/HIVE-15617 > Project: Hive > Issue Type: Sub-task > Components: PTF-Windowing >Affects Versions: 1.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-15617.1.patch > > > Similar to HIVE-15520, we need to improve the performance for avg(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15579) Support HADOOP_PROXY_USER for secure impersonation in hive metastore client
[ https://issues.apache.org/jira/browse/HIVE-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nanda kumar updated HIVE-15579: --- Attachment: HIVE-15579.001.patch > Support HADOOP_PROXY_USER for secure impersonation in hive metastore client > --- > > Key: HIVE-15579 > URL: https://issues.apache.org/jira/browse/HIVE-15579 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Nanda kumar > Attachments: HIVE-15579.000.patch, HIVE-15579.001.patch > > > Hadoop clients support HADOOP_PROXY_USER for secure impersonation. It would > be useful to have similar feature for hive metastore client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15579) Support HADOOP_PROXY_USER for secure impersonation in hive metastore client
[
https://issues.apache.org/jira/browse/HIVE-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826246#comment-15826246
]
Nanda kumar commented on HIVE-15579:
[~thejas], thanks for the review.
Incorporated the review comments in [^HIVE-15579.001.patch].
MetaException doesn't support (String, Exception), so added the following logic
{code}
catch (Exception e) {
LOG.error("Error while setting delegation token for " + proxyUser + "
user.");
if(e instanceof MetaException) {
throw (MetaException)e;
} else {
throw new MetaException(e.getMessage());
}
{code}
Please review.
> Support HADOOP_PROXY_USER for secure impersonation in hive metastore client
> ---
>
> Key: HIVE-15579
> URL: https://issues.apache.org/jira/browse/HIVE-15579
> Project: Hive
> Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Nanda kumar
> Attachments: HIVE-15579.000.patch, HIVE-15579.001.patch
>
>
> Hadoop clients support HADOOP_PROXY_USER for secure impersonation. It would
> be useful to have similar feature for hive metastore client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15642) Replicate Insert Overwrites
[ https://issues.apache.org/jira/browse/HIVE-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15642: Attachment: HIVE-15648.1.patch Attaching a patch w/o new test (will add tests after HIVE-15478 is in) > Replicate Insert Overwrites > --- > > Key: HIVE-15642 > URL: https://issues.apache.org/jira/browse/HIVE-15642 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > > 1. Insert Overwrites to a new partition should not capture new files as part > of insert event but instead use the subsequent add partition event to capture > the files + checksums. > 2. Insert Overwrites to an existing partition should capture new files as > part of the insert event. > This will need changes from HIVE-15478 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15642) Replicate Insert Overwrites
[ https://issues.apache.org/jira/browse/HIVE-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15642: Status: Patch Available (was: Open) > Replicate Insert Overwrites > --- > > Key: HIVE-15642 > URL: https://issues.apache.org/jira/browse/HIVE-15642 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > > 1. Insert Overwrites to a new partition should not capture new files as part > of insert event but instead use the subsequent add partition event to capture > the files + checksums. > 2. Insert Overwrites to an existing partition should capture new files as > part of the insert event. > This will need changes from HIVE-15478 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15642) Replicate Insert Overwrites
[ https://issues.apache.org/jira/browse/HIVE-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15642: Attachment: (was: HIVE-15648.1.patch) > Replicate Insert Overwrites > --- > > Key: HIVE-15642 > URL: https://issues.apache.org/jira/browse/HIVE-15642 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > > 1. Insert Overwrites to a new partition should not capture new files as part > of insert event but instead use the subsequent add partition event to capture > the files + checksums. > 2. Insert Overwrites to an existing partition should capture new files as > part of the insert event. > This will need changes from HIVE-15478 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15642) Replicate Insert Overwrites
[ https://issues.apache.org/jira/browse/HIVE-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15642: Attachment: HIVE-15642.1.patch > Replicate Insert Overwrites > --- > > Key: HIVE-15642 > URL: https://issues.apache.org/jira/browse/HIVE-15642 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-15642.1.patch > > > 1. Insert Overwrites to a new partition should not capture new files as part > of insert event but instead use the subsequent add partition event to capture > the files + checksums. > 2. Insert Overwrites to an existing partition should capture new files as > part of the insert event. > This will need changes from HIVE-15478 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15489) Alternatively use table scan stats for HoS
[ https://issues.apache.org/jira/browse/HIVE-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-15489: Attachment: (was: HIVE-15428.2.patch) > Alternatively use table scan stats for HoS > -- > > Key: HIVE-15489 > URL: https://issues.apache.org/jira/browse/HIVE-15489 > Project: Hive > Issue Type: Improvement > Components: Spark, Statistics >Affects Versions: 2.2.0 >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-15489.1.patch, HIVE-15489.wip.patch > > > For MapJoin in HoS, we should provide an option to only use stats in the TS > rather than the populated stats in each of the join branch. This could be > pretty conservative but more reliable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15489) Alternatively use table scan stats for HoS
[ https://issues.apache.org/jira/browse/HIVE-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-15489: Attachment: HIVE-15489.1.patch > Alternatively use table scan stats for HoS > -- > > Key: HIVE-15489 > URL: https://issues.apache.org/jira/browse/HIVE-15489 > Project: Hive > Issue Type: Improvement > Components: Spark, Statistics >Affects Versions: 2.2.0 >Reporter: Chao Sun >Assignee: Chao Sun > Attachments: HIVE-15489.1.patch, HIVE-15489.wip.patch > > > For MapJoin in HoS, we should provide an option to only use stats in the TS > rather than the populated stats in each of the join branch. This could be > pretty conservative but more reliable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15618) Change hive-blobstore tests to run with Tez by default
[ https://issues.apache.org/jira/browse/HIVE-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Poepping updated HIVE-15618: --- Status: Open (was: Patch Available) > Change hive-blobstore tests to run with Tez by default > -- > > Key: HIVE-15618 > URL: https://issues.apache.org/jira/browse/HIVE-15618 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-15618.patch > > > Ever since the upgrade to Hive 2, Tez has been the default execution engine > for Hive. To match that fact, it makes sense to run our tests against Tez, > rather than MR. This should more fully validate functionality against what we > consider to be Hive defaults. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15618) Change hive-blobstore tests to run with Tez by default
[ https://issues.apache.org/jira/browse/HIVE-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Poepping updated HIVE-15618: --- Status: Patch Available (was: Open) resubmitting patch, because it was never picked up by QA > Change hive-blobstore tests to run with Tez by default > -- > > Key: HIVE-15618 > URL: https://issues.apache.org/jira/browse/HIVE-15618 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-15618.patch > > > Ever since the upgrade to Hive 2, Tez has been the default execution engine > for Hive. To match that fact, it makes sense to run our tests against Tez, > rather than MR. This should more fully validate functionality against what we > consider to be Hive defaults. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15576) Fix bug in QTestUtil where lines after a partial mask will not be masked
[ https://issues.apache.org/jira/browse/HIVE-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826498#comment-15826498 ] Thomas Poepping commented on HIVE-15576: Ping, help? [~sershe] I see you watched this issue, do you mind taking a look at the patch? > Fix bug in QTestUtil where lines after a partial mask will not be masked > > > Key: HIVE-15576 > URL: https://issues.apache.org/jira/browse/HIVE-15576 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-15576.patch > > > If the qfile output of a qtest contains two maskable lines right after one > another, where the first contains a partial match candidate, the second line > will not be evaluated for masking. This patch fixes that bug by disregarding > whether a partial mask was found in the previous line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15646) Column level lineage is not available for table Views
[ https://issues.apache.org/jira/browse/HIVE-15646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826504#comment-15826504 ] Ashutosh Chauhan commented on HIVE-15646: - +1 pending tests > Column level lineage is not available for table Views > - > > Key: HIVE-15646 > URL: https://issues.apache.org/jira/browse/HIVE-15646 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-15646.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15472) JDBC: Standalone jar is missing ZK dependencies
[
https://issues.apache.org/jira/browse/HIVE-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Li updated HIVE-15472:
--
Status: Open (was: Patch Available)
> JDBC: Standalone jar is missing ZK dependencies
> ---
>
> Key: HIVE-15472
> URL: https://issues.apache.org/jira/browse/HIVE-15472
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Tao Li
> Attachments: HIVE-15472.1.patch, HIVE-15472.2.patch
>
>
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/curator/RetryPolicy
> at org.apache.hive.jdbc.Utils.configureConnParams(Utils.java:514)
> at org.apache.hive.jdbc.Utils.parseURL(Utils.java:434)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:132)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:247)
> at JDBCExecutor.getConnection(JDBCExecutor.java:65)
> at JDBCExecutor.executeStatement(JDBCExecutor.java:104)
> at JDBCExecutor.executeSQLFile(JDBCExecutor.java:81)
> at JDBCExecutor.main(JDBCExecutor.java:183)
> Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15472) JDBC: Standalone jar is missing ZK dependencies
[
https://issues.apache.org/jira/browse/HIVE-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Li updated HIVE-15472:
--
Attachment: HIVE-15472.2.patch
> JDBC: Standalone jar is missing ZK dependencies
> ---
>
> Key: HIVE-15472
> URL: https://issues.apache.org/jira/browse/HIVE-15472
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Tao Li
> Attachments: HIVE-15472.1.patch, HIVE-15472.2.patch
>
>
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/curator/RetryPolicy
> at org.apache.hive.jdbc.Utils.configureConnParams(Utils.java:514)
> at org.apache.hive.jdbc.Utils.parseURL(Utils.java:434)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:132)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:247)
> at JDBCExecutor.getConnection(JDBCExecutor.java:65)
> at JDBCExecutor.executeStatement(JDBCExecutor.java:104)
> at JDBCExecutor.executeSQLFile(JDBCExecutor.java:81)
> at JDBCExecutor.main(JDBCExecutor.java:183)
> Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15472) JDBC: Standalone jar is missing ZK dependencies
[
https://issues.apache.org/jira/browse/HIVE-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Li updated HIVE-15472:
--
Attachment: (was: HIVE-15472.2.patch)
> JDBC: Standalone jar is missing ZK dependencies
> ---
>
> Key: HIVE-15472
> URL: https://issues.apache.org/jira/browse/HIVE-15472
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Tao Li
> Attachments: HIVE-15472.1.patch, HIVE-15472.2.patch
>
>
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/curator/RetryPolicy
> at org.apache.hive.jdbc.Utils.configureConnParams(Utils.java:514)
> at org.apache.hive.jdbc.Utils.parseURL(Utils.java:434)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:132)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:247)
> at JDBCExecutor.getConnection(JDBCExecutor.java:65)
> at JDBCExecutor.executeStatement(JDBCExecutor.java:104)
> at JDBCExecutor.executeSQLFile(JDBCExecutor.java:81)
> at JDBCExecutor.main(JDBCExecutor.java:183)
> Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15472) JDBC: Standalone jar is missing ZK dependencies
[
https://issues.apache.org/jira/browse/HIVE-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Li updated HIVE-15472:
--
Status: Patch Available (was: Open)
> JDBC: Standalone jar is missing ZK dependencies
> ---
>
> Key: HIVE-15472
> URL: https://issues.apache.org/jira/browse/HIVE-15472
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Tao Li
> Attachments: HIVE-15472.1.patch, HIVE-15472.2.patch
>
>
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/curator/RetryPolicy
> at org.apache.hive.jdbc.Utils.configureConnParams(Utils.java:514)
> at org.apache.hive.jdbc.Utils.parseURL(Utils.java:434)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:132)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:247)
> at JDBCExecutor.getConnection(JDBCExecutor.java:65)
> at JDBCExecutor.executeStatement(JDBCExecutor.java:104)
> at JDBCExecutor.executeSQLFile(JDBCExecutor.java:81)
> at JDBCExecutor.main(JDBCExecutor.java:183)
> Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13014) RetryingMetaStoreClient is retrying too aggresievley
[ https://issues.apache.org/jira/browse/HIVE-13014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826572#comment-15826572 ] Eugene Koifman commented on HIVE-13014: --- The patch makes a few methods safe to retry (which were not so before) and annotates others to indicate retry semantics The worst case to avoid is when server side op succeeds (and commits against the metastore RDBMS) but the remote caller doesn't know this and retries an op that cannot be retried. [~alangates] could you review please > RetryingMetaStoreClient is retrying too aggresievley > > > Key: HIVE-13014 > URL: https://issues.apache.org/jira/browse/HIVE-13014 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-13014.01.patch, HIVE-13014.02.patch, > HIVE-13014.03.patch > > > Not all metastore operations are idempotent. For example, commit_txn() > consists of > 1. request from client to server > 2. server action > 3. ack to client > If network connection is broken after (or during) 2 but before 3 happens, > RetryingMetastoreClient will retry the operation thus causing an attempt to > commit the same txn twice (sometimes in concurrently) > The 2nd attempt is guaranteed to fail and thus return an error to the caller > (which doesn't know the operation is being retried), while the first attempt > has actually succeeded. Thus the caller thinks commit failed and will likely > attempt to redo the transactions - not what we want in most cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15576) Fix bug in QTestUtil where lines after a partial mask will not be masked
[ https://issues.apache.org/jira/browse/HIVE-15576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826591#comment-15826591 ] Sergey Shelukhin commented on HIVE-15576: - +1 will commit later today if noone objects > Fix bug in QTestUtil where lines after a partial mask will not be masked > > > Key: HIVE-15576 > URL: https://issues.apache.org/jira/browse/HIVE-15576 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-15576.patch > > > If the qfile output of a qtest contains two maskable lines right after one > another, where the first contains a partial match candidate, the second line > will not be evaluated for masking. This patch fixes that bug by disregarding > whether a partial mask was found in the previous line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14303) CommonJoinOperator.checkAndGenObject should return directly to avoid NPE if ExecReducer.close is called twice.
[
https://issues.apache.org/jira/browse/HIVE-14303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-14303:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to master. Thanks [~zxu] for the patch.
> CommonJoinOperator.checkAndGenObject should return directly to avoid NPE if
> ExecReducer.close is called twice.
> --
>
> Key: HIVE-14303
> URL: https://issues.apache.org/jira/browse/HIVE-14303
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
> Fix For: 2.2.0
>
> Attachments: HIVE-14303.0.patch, HIVE-14303.1.patch,
> HIVE-14303.2.patch
>
>
> CommonJoinOperator.checkAndGenObject should return directly (after
> {{CommonJoinOperator.closeOp}} was called ) to avoid NPE if ExecReducer.close
> is called twice. ExecReducer.close implements Closeable interface and
> ExecReducer.close can be called multiple time. We saw the following NPE which
> hide the real exception due to this bug.
> {code}
> Error: java.lang.RuntimeException: Hive Runtime Error while closing
> operators: null
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296)
> at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:718)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284)
> ... 8 more
> {code}
> The code from ReduceTask.runOldReducer:
> {code}
> reducer.close(); //line 453
> reducer = null;
>
> out.close(reporter);
> out = null;
> } finally {
> IOUtils.cleanup(LOG, reducer);// line 459
> closeQuietly(out, reporter);
> }
> {code}
> Based on the above stack trace and code, reducer.close() is called twice
> because the exception happened when reducer.close() is called for the first
> time at line 453, the code exit before reducer was set to null.
> NullPointerException is triggered when reducer.close() is called for the
> second time in IOUtils.cleanup at line 459. NullPointerException hide the
> real exception which happened when reducer.close() is called for the first
> time at line 453.
> The reason for NPE is:
> The first reducer.close called CommonJoinOperator.closeOp which clear
> {{storage}}
> {code}
> Arrays.fill(storage, null);
> {code}
> the second reduce.close generated NPE due to null {{storage[alias]}} which is
> set to null by first reducer.close.
> The following reducer log can give more proof:
> {code}
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.JoinOperator: 0 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.JoinOperator: SKEWJOINFOLLOWUPJOBS:0
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.SelectOperator: 2 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.SelectOperator: 3 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.FileSinkOperator: 4 finished. closing...
> 2016-07-14 22:24:51,016 INFO [main]
> org.apache.hadoop.hive.ql.exec.FileSinkOperator: FS[4]: records written -
> 53466
> 2016-07-14 22:25:11,555 ERROR [main] ExecReducer: Hit error while closing
> operators - failing tree
> 2016-07-14 22:25:11,649 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.lang.RuntimeException: Hive Runtime Error
> while closing operators: null
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:296)
> at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.ja
[jira] [Commented] (HIVE-15629) Set DDLTask’s exception with its subtask’s exception
[ https://issues.apache.org/jira/browse/HIVE-15629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826604#comment-15826604 ] Jimmy Xiang commented on HIVE-15629: +1 > Set DDLTask’s exception with its subtask’s exception > > > Key: HIVE-15629 > URL: https://issues.apache.org/jira/browse/HIVE-15629 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 2.2.0 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-15629.000.patch > > > Set DDLTask’s exception with its subtask’s exception, So the exception from > subtask in DDLTask can be propagated to TaskRunner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15648) Hive throws compilation error due to $powermock.version not being present in root pom
[ https://issues.apache.org/jira/browse/HIVE-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826615#comment-15826615 ] Thejas M Nair commented on HIVE-15648: -- +1 Please go ahead and commit. > Hive throws compilation error due to $powermock.version not being present in > root pom > - > > Key: HIVE-15648 > URL: https://issues.apache.org/jira/browse/HIVE-15648 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-15648.1.patch > > > Looks like caused by HIVE-15550 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15645) Tez session pool may restart sessions in a wrong queue
[ https://issues.apache.org/jira/browse/HIVE-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826617#comment-15826617 ] Gunther Hagleitner commented on HIVE-15645: --- Your comment says: // There's some bogus code that can modify the queue name. Force-set it for pool sessions. This fix sounds like a guess. Is there an actual test case to show what's being fixed? If there is bogus code, can we fix that? > Tez session pool may restart sessions in a wrong queue > -- > > Key: HIVE-15645 > URL: https://issues.apache.org/jira/browse/HIVE-15645 > Project: Hive > Issue Type: Bug >Reporter: Carter Shanklin >Assignee: Sergey Shelukhin > Attachments: HIVE-15645.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15645) Tez session pool may restart sessions in a wrong queue
[ https://issues.apache.org/jira/browse/HIVE-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826628#comment-15826628 ] Sergey Shelukhin commented on HIVE-15645: - We had a repro on some cluster that indicates that the patch will fix the problem. It has to do with config being out of sync with the property. First session gets config and property correct, but something (I am pretty sure it's the unset in open path) resets the config. Then the 2nd session (after expiration) gets the property correct but the config is not set, so it logs as if it is going to correct queue but goes to wrong queue, which is what we have observed for a specific session. The field is also reset to null from conf (in a place where I added the warn log), after the log statement about the queue. The 3rd session (after the 2nd expiration) logs null queue (because the field is also null now), and goes to the wrong queue, as does every one after that. So, for pool sessions we set the session into conf every time now. I also added a warn log for the future, and a null check cause we never expect null queue for pool sessions. To fix this properly the separation of pool and non-pool sessions that was started at some point needs to be completed, but that's a major refactoring, not a bugfix. > Tez session pool may restart sessions in a wrong queue > -- > > Key: HIVE-15645 > URL: https://issues.apache.org/jira/browse/HIVE-15645 > Project: Hive > Issue Type: Bug >Reporter: Carter Shanklin >Assignee: Sergey Shelukhin > Attachments: HIVE-15645.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15645) Tez session pool may restart sessions in a wrong queue
[ https://issues.apache.org/jira/browse/HIVE-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826628#comment-15826628 ] Sergey Shelukhin edited comment on HIVE-15645 at 1/17/17 7:11 PM: -- We had a repro on some cluster that indicates that the patch will fix the problem. It has to do with config being out of sync with the property. First session gets config and property correct, but something (I am pretty sure it's the unset in open path) resets the config. Then the 2nd session (after expiration) gets the property correct but the config is not set, so it logs as if it is going to correct queue but goes to a wrong (default) queue, which is what we have observed for a specific session in the cluster. The field is also reset to null from conf (in a place where I added the warn log), after the log statement about the queue. The 3rd session (after the 2nd expiration) logs null queue (because the field is also null now), and goes to the wrong queue, as does every one after that. So, for pool sessions we set the session into conf every time now. I also added a warn log for the future, and a null check cause we never expect null queue for pool sessions. To fix this properly the separation of pool and non-pool sessions that was started at some point needs to be completed, but that's a major refactoring, not a bugfix. was (Author: sershe): We had a repro on some cluster that indicates that the patch will fix the problem. It has to do with config being out of sync with the property. First session gets config and property correct, but something (I am pretty sure it's the unset in open path) resets the config. Then the 2nd session (after expiration) gets the property correct but the config is not set, so it logs as if it is going to correct queue but goes to wrong queue, which is what we have observed for a specific session. The field is also reset to null from conf (in a place where I added the warn log), after the log statement about the queue. The 3rd session (after the 2nd expiration) logs null queue (because the field is also null now), and goes to the wrong queue, as does every one after that. So, for pool sessions we set the session into conf every time now. I also added a warn log for the future, and a null check cause we never expect null queue for pool sessions. To fix this properly the separation of pool and non-pool sessions that was started at some point needs to be completed, but that's a major refactoring, not a bugfix. > Tez session pool may restart sessions in a wrong queue > -- > > Key: HIVE-15645 > URL: https://issues.apache.org/jira/browse/HIVE-15645 > Project: Hive > Issue Type: Bug >Reporter: Carter Shanklin >Assignee: Sergey Shelukhin > Attachments: HIVE-15645.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15648) Hive throws compilation error due to $powermock.version not being present in root pom
[ https://issues.apache.org/jira/browse/HIVE-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15648: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed. Thanks [~thejas] > Hive throws compilation error due to $powermock.version not being present in > root pom > - > > Key: HIVE-15648 > URL: https://issues.apache.org/jira/browse/HIVE-15648 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Fix For: 2.2.0 > > Attachments: HIVE-15648.1.patch > > > Looks like caused by HIVE-15550 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15645) Tez session pool may restart sessions in a wrong queue
[ https://issues.apache.org/jira/browse/HIVE-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826692#comment-15826692 ] Gunther Hagleitner commented on HIVE-15645: --- This still might be one step forward two steps back. How do you know that clearing the conf wasn't done for some specific reason? Also, the exception nukes a case that used to work - what if that was done intentionally? > Tez session pool may restart sessions in a wrong queue > -- > > Key: HIVE-15645 > URL: https://issues.apache.org/jira/browse/HIVE-15645 > Project: Hive > Issue Type: Bug >Reporter: Carter Shanklin >Assignee: Sergey Shelukhin > Attachments: HIVE-15645.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15645) Tez session pool may restart sessions in a wrong queue
[ https://issues.apache.org/jira/browse/HIVE-15645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826708#comment-15826708 ] Sergey Shelukhin commented on HIVE-15645: - The unsetting of the config is done for sessions with queue specified by the user, which is not something supported by pool sessions. The exception case is to manifest future bugs (if any) better - it was not supposed to work, and thus worked improperly... > Tez session pool may restart sessions in a wrong queue > -- > > Key: HIVE-15645 > URL: https://issues.apache.org/jira/browse/HIVE-15645 > Project: Hive > Issue Type: Bug >Reporter: Carter Shanklin >Assignee: Sergey Shelukhin > Attachments: HIVE-15645.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15650) LLAP: Set perflogger to DEBUG level for llap daemons
[ https://issues.apache.org/jira/browse/HIVE-15650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-15650: - Status: Patch Available (was: Open) > LLAP: Set perflogger to DEBUG level for llap daemons > > > Key: HIVE-15650 > URL: https://issues.apache.org/jira/browse/HIVE-15650 > Project: Hive > Issue Type: Bug > Components: llap, Logging >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran > Attachments: HIVE-15650.1.patch > > > During Hive2 dev, the PerfLogger was moved to DEBUG levels only making it > impossible to debug timings from LLAP logs without manually editing > log4j2.properties and redeploying LLAP. > Enable PerfLogger by default on LLAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15650) LLAP: Set perflogger to DEBUG level for llap daemons
[ https://issues.apache.org/jira/browse/HIVE-15650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-15650: - Attachment: HIVE-15650.1.patch > LLAP: Set perflogger to DEBUG level for llap daemons > > > Key: HIVE-15650 > URL: https://issues.apache.org/jira/browse/HIVE-15650 > Project: Hive > Issue Type: Bug > Components: llap, Logging >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran > Attachments: HIVE-15650.1.patch > > > During Hive2 dev, the PerfLogger was moved to DEBUG levels only making it > impossible to debug timings from LLAP logs without manually editing > log4j2.properties and redeploying LLAP. > Enable PerfLogger by default on LLAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15650) LLAP: Set perflogger to DEBUG level for llap daemons
[ https://issues.apache.org/jira/browse/HIVE-15650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826873#comment-15826873 ] Prasanth Jayachandran commented on HIVE-15650: -- [~gopalv] can you please review this patch? > LLAP: Set perflogger to DEBUG level for llap daemons > > > Key: HIVE-15650 > URL: https://issues.apache.org/jira/browse/HIVE-15650 > Project: Hive > Issue Type: Bug > Components: llap, Logging >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran > Attachments: HIVE-15650.1.patch > > > During Hive2 dev, the PerfLogger was moved to DEBUG levels only making it > impossible to debug timings from LLAP logs without manually editing > log4j2.properties and redeploying LLAP. > Enable PerfLogger by default on LLAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15643) remove use of default charset in FastHiveDecimal
[ https://issues.apache.org/jira/browse/HIVE-15643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826882#comment-15826882 ] Prasanth Jayachandran commented on HIVE-15643: -- [~owen.omalley] changing it to UTF8 would break bloom filters because of hashcode difference. We might need a separate stream for UTF8, make the readers use UTF8 by default and for old files use default charset stream. > remove use of default charset in FastHiveDecimal > > > Key: HIVE-15643 > URL: https://issues.apache.org/jira/browse/HIVE-15643 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Edward Capriolo > > HIVE-15335 introduced some new uses of String.getBytes(), which uses the > default char set. These need to be replaced with the version that always uses > UTF8. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-15573) Vectorization: ACID shuffle ReduceSink is not specialized
[
https://issues.apache.org/jira/browse/HIVE-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-15573:
-
Assignee: Eugene Koifman
> Vectorization: ACID shuffle ReduceSink is not specialized
> --
>
> Key: HIVE-15573
> URL: https://issues.apache.org/jira/browse/HIVE-15573
> Project: Hive
> Issue Type: Improvement
> Components: Transactions, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Eugene Koifman
> Attachments: screenshot-1.png
>
>
> The ACID shuffle disabled murmur hash for the shuffle, due to the bucketing
> requirements demanding the writable hashcode for the shuffles.
> {code}
> boolean useUniformHash = desc.getReducerTraits().contains(UNIFORM);
> if (!useUniformHash) {
> return false;
> }
> {code}
> This check protects the fast ReduceSink ops from being used in ACID inserts.
> A specialized case for the following pattern will make ACID insert much
> faster.
> {code}
> Reduce Output Operator
> sort order:
> Map-reduce partition columns: _col0 (type: bigint)
> value expressions:
> {code}
> !screenshot-1.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14926) Keep Schema in consistent state where schemaTool fails or succeeds.
[ https://issues.apache.org/jira/browse/HIVE-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-14926: Resolution: Won't Fix Assignee: (was: Aihua Xu) Status: Resolved (was: Patch Available) > Keep Schema in consistent state where schemaTool fails or succeeds. > - > > Key: HIVE-14926 > URL: https://issues.apache.org/jira/browse/HIVE-14926 > Project: Hive > Issue Type: Sub-task > Components: Database/Schema >Reporter: Aihua Xu > Attachments: HIVE-14926.1.patch, HIVE-14926.2.patch > > > SchemaTool uses autocommit right now when executing the upgrade or init > scripts. Seems we should use database transaction to commit or roll back to > keep schema consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15534) Update db/table repl.last.id at the end of REPL LOAD of a batch of events
[
https://issues.apache.org/jira/browse/HIVE-15534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-15534:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Thanks, [~daijy], committed to master.
> Update db/table repl.last.id at the end of REPL LOAD of a batch of events
> -
>
> Key: HIVE-15534
> URL: https://issues.apache.org/jira/browse/HIVE-15534
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Fix For: 2.2.0
>
> Attachments: HIVE-15534.patch
>
>
> Tracking TODO task in ReplSemanticAnalyzer :
> {noformat}
> // TODO : Over here, we need to track a
> Map for every db updated
> // and update repl.last.id for each, if this is a wh-level load, and
> if it is a db-level load,
> // then a single repl.last.id update, and if this is a tbl-lvl load
> which does not alter the
> // table itself, we'll need to update repl.last.id for that as well.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Liew updated HIVE-13680: -- Attachment: HIVE-13680.7.patch Attached a new patch with cleaner negotiation and support for versioned plug-ins. > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta >Assignee: Kevin Liew > Attachments: HIVE-13680.2.patch, HIVE-13680.3.patch, > HIVE-13680.4.patch, HIVE-13680.6.patch, HIVE-13680.7.patch, HIVE-13680.patch, > proposal.pdf, SnappyCompDe.zip > > > With HIVE-12049 in, we can provide an option to compress ResultSets before > writing to disk. The user can specify a compression library via a config > param which can be used in the tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14946) Optimizations
[ https://issues.apache.org/jira/browse/HIVE-14946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-14946: - Assignee: Eugene Koifman > Optimizations > - > > Key: HIVE-14946 > URL: https://issues.apache.org/jira/browse/HIVE-14946 > Project: Hive > Issue Type: Sub-task > Components: Query Planning, Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > For example, if there is only WHEN NOT MATCHED clause, the base generating > expression can be an INNER JOIN > Generally, we should make sure the optimizer is able to work with the plan > for Merge statement > Various WHEN clauses can have "extra" predicates. In some cases they may be > pushable. > The "source" can be an arbitrary expression - in particular it may include > joins which together with the join introduced by Merge itself may need to be > reordered. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-14946) Optimizations
[ https://issues.apache.org/jira/browse/HIVE-14946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman resolved HIVE-14946. --- Resolution: Duplicate Fix Version/s: 2.2.0 addressed by HIVE-15539 > Optimizations > - > > Key: HIVE-14946 > URL: https://issues.apache.org/jira/browse/HIVE-14946 > Project: Hive > Issue Type: Sub-task > Components: Query Planning, Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 2.2.0 > > > For example, if there is only WHEN NOT MATCHED clause, the base generating > expression can be an INNER JOIN > Generally, we should make sure the optimizer is able to work with the plan > for Merge statement > Various WHEN clauses can have "extra" predicates. In some cases they may be > pushable. > The "source" can be an arbitrary expression - in particular it may include > joins which together with the join introduced by Merge itself may need to be > reordered. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15550) fix arglist logging in schematool
[ https://issues.apache.org/jira/browse/HIVE-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827029#comment-15827029 ] Thejas M Nair commented on HIVE-15550: -- Thanks [~leftylev] for pointing the mistake and the example! I have now updated the errata.txt . > fix arglist logging in schematool > - > > Key: HIVE-15550 > URL: https://issues.apache.org/jira/browse/HIVE-15550 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.1.1 >Reporter: anishek >Assignee: anishek >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-15550.1.patch > > > In DEBUG mode schemaTool prints the password to log file. > This is also seen if the user includes --verbose option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15649) LLAP IO may NPE on all-column read
[ https://issues.apache.org/jira/browse/HIVE-15649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15649: Attachment: HIVE-15649.patch Added a test too, however on MiniLlap/Local it doesn't repro the issue that we've seen on some cluster where analyze table ... for columns resulted in null columnIds. [~prasanth_j] can you take a look? [~mmccline] FYI I think some schema evolution paths may also assume non-null column list... e.g. buildConversionFileTypesArray seems to assume in isOk path that readerIncludes are not null even though they could be. > LLAP IO may NPE on all-column read > -- > > Key: HIVE-15649 > URL: https://issues.apache.org/jira/browse/HIVE-15649 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15649.patch > > > It seems like very few paths use READ_ALL_COLUMNS config, but some do. LLAP > IO doesn't account for that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15649) LLAP IO may NPE on all-column read
[ https://issues.apache.org/jira/browse/HIVE-15649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15649: Status: Patch Available (was: Open) > LLAP IO may NPE on all-column read > -- > > Key: HIVE-15649 > URL: https://issues.apache.org/jira/browse/HIVE-15649 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-15649.patch > > > It seems like very few paths use READ_ALL_COLUMNS config, but some do. LLAP > IO doesn't account for that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15621) Use Hive's own JvmPauseMonitor instead of Hadoop's in LLAP
[ https://issues.apache.org/jira/browse/HIVE-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15621: - Status: Open (was: Patch Available) > Use Hive's own JvmPauseMonitor instead of Hadoop's in LLAP > -- > > Key: HIVE-15621 > URL: https://issues.apache.org/jira/browse/HIVE-15621 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15621.1.patch, HIVE-15621.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15621) Use Hive's own JvmPauseMonitor instead of Hadoop's in LLAP
[ https://issues.apache.org/jira/browse/HIVE-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15621: - Attachment: HIVE-15621.3.patch > Use Hive's own JvmPauseMonitor instead of Hadoop's in LLAP > -- > > Key: HIVE-15621 > URL: https://issues.apache.org/jira/browse/HIVE-15621 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15621.1.patch, HIVE-15621.2.patch, > HIVE-15621.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15621) Use Hive's own JvmPauseMonitor instead of Hadoop's in LLAP
[ https://issues.apache.org/jira/browse/HIVE-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15621: - Status: Patch Available (was: Open) > Use Hive's own JvmPauseMonitor instead of Hadoop's in LLAP > -- > > Key: HIVE-15621 > URL: https://issues.apache.org/jira/browse/HIVE-15621 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15621.1.patch, HIVE-15621.2.patch, > HIVE-15621.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15579) Support HADOOP_PROXY_USER for secure impersonation in hive metastore client
[
https://issues.apache.org/jira/browse/HIVE-15579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827088#comment-15827088
]
Thejas M Nair commented on HIVE-15579:
--
Can you retain the exception in log message like previous patch ?
ie use - LOG.error("Error while setting delegation token for " + proxyUser, e);
> Support HADOOP_PROXY_USER for secure impersonation in hive metastore client
> ---
>
> Key: HIVE-15579
> URL: https://issues.apache.org/jira/browse/HIVE-15579
> Project: Hive
> Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Nanda kumar
> Attachments: HIVE-15579.000.patch, HIVE-15579.001.patch
>
>
> Hadoop clients support HADOOP_PROXY_USER for secure impersonation. It would
> be useful to have similar feature for hive metastore client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15623) Use customized version of netty for llap
[ https://issues.apache.org/jira/browse/HIVE-15623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15623: - Status: Patch Available (was: Open) > Use customized version of netty for llap > > > Key: HIVE-15623 > URL: https://issues.apache.org/jira/browse/HIVE-15623 > Project: Hive > Issue Type: Task > Components: llap >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-15623.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15627) Make hive.vectorized.adaptor.usage.mode=all vectorize all UDFs not just those in supportedGenericUDFs
[ https://issues.apache.org/jira/browse/HIVE-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15627: Status: Patch Available (was: In Progress) > Make hive.vectorized.adaptor.usage.mode=all vectorize all UDFs not just those > in supportedGenericUDFs > - > > Key: HIVE-15627 > URL: https://issues.apache.org/jira/browse/HIVE-15627 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15627.01.patch, HIVE-15627.02.patch, > HIVE-15627.03.patch, HIVE-15627.04.patch, HIVE-15627.05.patch > > > Missed this when doing HIVE-14336. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15588) Vectorization: Fix deallocation of scratch columns in VectorUDFCoalesce, etc to prevent wrong reuse
[ https://issues.apache.org/jira/browse/HIVE-15588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-15588: Status: Patch Available (was: In Progress) > Vectorization: Fix deallocation of scratch columns in VectorUDFCoalesce, etc > to prevent wrong reuse > --- > > Key: HIVE-15588 > URL: https://issues.apache.org/jira/browse/HIVE-15588 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-15588.01.patch, HIVE-15588.02.patch, > HIVE-15588.03.patch, HIVE-15588.04.patch, HIVE-15588.05.patch > > > Make sure we don't deallocate a scratch column too quickly and cause result > corruption due to scratch column reuse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15534) Update db/table repl.last.id at the end of REPL LOAD of a batch of events
[
https://issues.apache.org/jira/browse/HIVE-15534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827143#comment-15827143
]
Sushanth Sowmyan commented on HIVE-15534:
-
[~daijy], I created HIVE-15652 to track the update after your review comments.
> Update db/table repl.last.id at the end of REPL LOAD of a batch of events
> -
>
> Key: HIVE-15534
> URL: https://issues.apache.org/jira/browse/HIVE-15534
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Fix For: 2.2.0
>
> Attachments: HIVE-15534.patch
>
>
> Tracking TODO task in ReplSemanticAnalyzer :
> {noformat}
> // TODO : Over here, we need to track a
> Map for every db updated
> // and update repl.last.id for each, if this is a wh-level load, and
> if it is a db-level load,
> // then a single repl.last.id update, and if this is a tbl-lvl load
> which does not alter the
> // table itself, we'll need to update repl.last.id for that as well.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14707) ACID: Insert shuffle sort-merges on blank KEY
[
https://issues.apache.org/jira/browse/HIVE-14707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-14707:
--
Attachment: HIVE-14707.21.patch
> ACID: Insert shuffle sort-merges on blank KEY
> -
>
> Key: HIVE-14707
> URL: https://issues.apache.org/jira/browse/HIVE-14707
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Eugene Koifman
> Attachments: HIVE-14707.01.patch, HIVE-14707.02.patch,
> HIVE-14707.03.patch, HIVE-14707.04.patch, HIVE-14707.05.patch,
> HIVE-14707.06.patch, HIVE-14707.08.patch, HIVE-14707.09.patch,
> HIVE-14707.10.patch, HIVE-14707.11.patch, HIVE-14707.13.patch,
> HIVE-14707.14.patch, HIVE-14707.16.patch, HIVE-14707.17.patch,
> HIVE-14707.18.patch, HIVE-14707.19.patch, HIVE-14707.19.patch,
> HIVE-14707.20.patch, HIVE-14707.21.patch
>
>
> The ACID insert codepath uses a sorted shuffle, while they key used for
> shuffle is always 0 bytes long.
> {code}
> hive (sales_acid)> explain insert into sales values(1, 2,
> '3400---009', 1, null);
> STAGE PLANS:
> Stage: Stage-1
> Tez
> DagId: gopal_20160906172626_80261c4c-79cc-4e02-87fe-3133be404e55:2
> Edges:
> Reducer 2 <- Map 1 (SIMPLE_EDGE)
> ...
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: values__tmp__table__2
> Statistics: Num rows: 1 Data size: 28 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: tmp_values_col1 (type: string),
> tmp_values_col2 (type: string), tmp_values_col3 (type: string),
> tmp_values_col4 (type: string), tmp_values_col5 (type: string)
> outputColumnNames: _col0, _col1, _col2, _col3, _col4
> Statistics: Num rows: 1 Data size: 28 Basic stats:
> COMPLETE Column stats: NONE
> Reduce Output Operator
> sort order:
> Map-reduce partition columns: UDFToLong(_col1) (type:
> bigint)
> Statistics: Num rows: 1 Data size: 28 Basic stats:
> COMPLETE Column stats: NONE
> value expressions: _col0 (type: string), _col1 (type:
> string), _col2 (type: string), _col3 (type: string), _col4 (type: string)
> Execution mode: vectorized, llap
> LLAP IO: no inputs
> {code}
> Note the missing "+" / "-" in the Sort Order fields.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13014) RetryingMetaStoreClient is retrying too aggresievley
[ https://issues.apache.org/jira/browse/HIVE-13014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13014: -- Attachment: HIVE-13014.04.patch > RetryingMetaStoreClient is retrying too aggresievley > > > Key: HIVE-13014 > URL: https://issues.apache.org/jira/browse/HIVE-13014 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-13014.01.patch, HIVE-13014.02.patch, > HIVE-13014.03.patch, HIVE-13014.04.patch > > > Not all metastore operations are idempotent. For example, commit_txn() > consists of > 1. request from client to server > 2. server action > 3. ack to client > If network connection is broken after (or during) 2 but before 3 happens, > RetryingMetastoreClient will retry the operation thus causing an attempt to > commit the same txn twice (sometimes in concurrently) > The 2nd attempt is guaranteed to fail and thus return an error to the caller > (which doesn't know the operation is being retried), while the first attempt > has actually succeeded. Thus the caller thinks commit failed and will likely > attempt to redo the transactions - not what we want in most cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15472) JDBC: Standalone jar is missing ZK dependencies
[
https://issues.apache.org/jira/browse/HIVE-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Li updated HIVE-15472:
--
Attachment: (was: HIVE-15472.2.patch)
> JDBC: Standalone jar is missing ZK dependencies
> ---
>
> Key: HIVE-15472
> URL: https://issues.apache.org/jira/browse/HIVE-15472
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Tao Li
> Attachments: HIVE-15472.1.patch
>
>
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/curator/RetryPolicy
> at org.apache.hive.jdbc.Utils.configureConnParams(Utils.java:514)
> at org.apache.hive.jdbc.Utils.parseURL(Utils.java:434)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:132)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:247)
> at JDBCExecutor.getConnection(JDBCExecutor.java:65)
> at JDBCExecutor.executeStatement(JDBCExecutor.java:104)
> at JDBCExecutor.executeSQLFile(JDBCExecutor.java:81)
> at JDBCExecutor.main(JDBCExecutor.java:183)
> Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15472) JDBC: Standalone jar is missing ZK dependencies
[
https://issues.apache.org/jira/browse/HIVE-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Li updated HIVE-15472:
--
Attachment: HIVE-15472.2.patch
> JDBC: Standalone jar is missing ZK dependencies
> ---
>
> Key: HIVE-15472
> URL: https://issues.apache.org/jira/browse/HIVE-15472
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Tao Li
> Attachments: HIVE-15472.1.patch, HIVE-15472.2.patch
>
>
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/curator/RetryPolicy
> at org.apache.hive.jdbc.Utils.configureConnParams(Utils.java:514)
> at org.apache.hive.jdbc.Utils.parseURL(Utils.java:434)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:132)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:247)
> at JDBCExecutor.getConnection(JDBCExecutor.java:65)
> at JDBCExecutor.executeStatement(JDBCExecutor.java:104)
> at JDBCExecutor.executeSQLFile(JDBCExecutor.java:81)
> at JDBCExecutor.main(JDBCExecutor.java:183)
> Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15652) Optimize(reduce) the number of alter calls made to fix repl.last.id
[ https://issues.apache.org/jira/browse/HIVE-15652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-15652: Attachment: HIVE-15652.patch Patch attached, this changes the total number of Alters to update repl.last.id across TestReplicationScenarios from 60 to 44. > Optimize(reduce) the number of alter calls made to fix repl.last.id > --- > > Key: HIVE-15652 > URL: https://issues.apache.org/jira/browse/HIVE-15652 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-15652.patch > > > Per code review from HIVE-15534, we might be doing alters to parent objects > to set repl.last.id when it is not necessary, since some future event might > make this alter redundant. > There are 3 cases where this might happen: > a) After a CREATE_TABLE event - any prior reference to that table does not > need an ALTER, since CREATE_TABLE will have a repl.last.id come with it. > b) After a DROP_TABLE event - any prior reference to that table is > irrelevant, and thus, no alter is needed. > c) After an ALTER_TABLE event, since that dump will itself do a metadata > update that will get the latest repl.last.id along with this event. > In each of these cases, we can remove the alter call needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15472) JDBC: Standalone jar is missing ZK dependencies
[
https://issues.apache.org/jira/browse/HIVE-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Li updated HIVE-15472:
--
Status: Open (was: Patch Available)
> JDBC: Standalone jar is missing ZK dependencies
> ---
>
> Key: HIVE-15472
> URL: https://issues.apache.org/jira/browse/HIVE-15472
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Tao Li
> Attachments: HIVE-15472.1.patch, HIVE-15472.2.patch
>
>
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/curator/RetryPolicy
> at org.apache.hive.jdbc.Utils.configureConnParams(Utils.java:514)
> at org.apache.hive.jdbc.Utils.parseURL(Utils.java:434)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:132)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:247)
> at JDBCExecutor.getConnection(JDBCExecutor.java:65)
> at JDBCExecutor.executeStatement(JDBCExecutor.java:104)
> at JDBCExecutor.executeSQLFile(JDBCExecutor.java:81)
> at JDBCExecutor.main(JDBCExecutor.java:183)
> Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15472) JDBC: Standalone jar is missing ZK dependencies
[
https://issues.apache.org/jira/browse/HIVE-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Li updated HIVE-15472:
--
Status: Patch Available (was: Open)
> JDBC: Standalone jar is missing ZK dependencies
> ---
>
> Key: HIVE-15472
> URL: https://issues.apache.org/jira/browse/HIVE-15472
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Tao Li
> Attachments: HIVE-15472.1.patch, HIVE-15472.2.patch
>
>
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/curator/RetryPolicy
> at org.apache.hive.jdbc.Utils.configureConnParams(Utils.java:514)
> at org.apache.hive.jdbc.Utils.parseURL(Utils.java:434)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:132)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:247)
> at JDBCExecutor.getConnection(JDBCExecutor.java:65)
> at JDBCExecutor.executeStatement(JDBCExecutor.java:104)
> at JDBCExecutor.executeSQLFile(JDBCExecutor.java:81)
> at JDBCExecutor.main(JDBCExecutor.java:183)
> Caused by: java.lang.ClassNotFoundException: org.apache.curator.RetryPolicy
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15478) Add file + checksum list for create table/partition during notification creation (whenever relevant)
[
https://issues.apache.org/jira/browse/HIVE-15478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827152#comment-15827152
]
Hive QA commented on HIVE-15478:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12847769/HIVE-15478.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 373 failed/errored test(s), 10789 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
org.apache.hadoop.hive.accumulo.TestAccumuloConnectionParameters.testMissingPassword
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloConnectionParameters.testMissingUserName
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testDropTableWithoutDeleteLeavesTableIntact
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testEmptyIteratorPushdownValue
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testExternalNonExistentTableFails
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testMissingColumnMappingFails
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testNonBooleanIteratorPushdownValue
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testNonExternalExistentTable
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testNonNullLocation
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testPreCreateTable
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testRollbackCreateTableDeletesExistentTable
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testRollbackCreateTableDoesntDeleteExternalExistentTable
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testRollbackCreateTableOnNonExistentTable
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testTableJobPropertiesCallsInputAndOutputMethods
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testTablePropertiesPassedToInputJobProperties
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testTablePropertiesPassedToOutputJobProperties
(batchId=165)
org.apache.hadoop.hive.accumulo.TestHiveAccumuloHelper.testISEIsPropagated
(batchId=165)
org.apache.hadoop.hive.accumulo.TestHiveAccumuloHelper.testISEIsPropagatedWithReflection
(batchId=165)
org.apache.hadoop.hive.accumulo.TestHiveAccumuloHelper.testTokenMerge
(batchId=165)
org.apache.hadoop.hive.accumulo.TestHiveAccumuloHelper.testTokenToConfFromUser
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableInputFormat.testConfigureAccumuloInputFormat
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableInputFormat.testConfigureAccumuloInputFormatWithAuthorizations
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableInputFormat.testConfigureAccumuloInputFormatWithEmptyColumns
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableInputFormat.testConfigureAccumuloInputFormatWithIterators
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableInputFormat.testConfigureMockAccumuloInputFormat
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableInputFormat.testIteratorNotInSplitsCompensation
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableOutputFormat.testBasicConfiguration
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableOutputFormat.testMockInstance
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableOutputFormat.testSaslConfiguration
(batchId=165)
org.apache.hadoop.hive.accumulo.predicate.TestAccumuloPredicateHandler.testEmptyListRangeGeneratorOutput
(batchId=165)
org.apache.hadoop.hive.accumulo.predicate.TestAccumuloPredicateHandler.testManyRangesGeneratorOutput
(batchId=165)
org.apache.hadoop.hive.accumulo.predicate.TestAccumuloPredicateHandler.testNullRangeGeneratorOutput
(batchId=165)
org.apache.hadoop.hive.accumulo.predicate.TestAccumuloPredicateHandler.testSingleRangeGeneratorOutput
(batchId=165)
org.apache.hadoop.hive.accumulo.serde.TestAccumuloRowSerializer.testBufferResetBeforeUse
(batchId=165)
org.apache.hadoop.hive.cli.TestCliDriverMethods.testThatCliDriverPrintsHeaderForCommandsWithSchema
(batchId=166)
org.apache.hadoop.hive.cli.TestCliDriverMethods.testThatCliDriverPrintsNoHeaderForCommandsWithNoSchema
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=148)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
(batchId=97)
org.apache.hadoop.
[jira] [Commented] (HIVE-10487) remove non-ISO restriction that projections in a union have identical column names
[ https://issues.apache.org/jira/browse/HIVE-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827154#comment-15827154 ] Pengcheng Xiong commented on HIVE-10487: This is not a problem any more on current hive master. I tried cbo=false and true. both of them work. intersect, except also work with cbo=true. > remove non-ISO restriction that projections in a union have identical column > names > -- > > Key: HIVE-10487 > URL: https://issues.apache.org/jira/browse/HIVE-10487 > Project: Hive > Issue Type: Improvement > Components: SQL >Affects Versions: 0.13.1 >Reporter: N Campbell >Priority: Critical > > While documented > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Union > an application should be able to perform a union query where the projections > are union compatible which does not include the projected column names being > identical which Hive imposes vs ISO-SQL 20xx. > i.e > rejected > select c1 from t1 union all select c2 from t2 > Schema of both sides of union should match. _u1-subquery2 > accepted > select c1 from t1 union all select c2 c1 from t2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-10487) remove non-ISO restriction that projections in a union have identical column names
[ https://issues.apache.org/jira/browse/HIVE-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong resolved HIVE-10487. Resolution: Not A Problem closed as not a problem. please feel free to reopen if it still exists. thanks. > remove non-ISO restriction that projections in a union have identical column > names > -- > > Key: HIVE-10487 > URL: https://issues.apache.org/jira/browse/HIVE-10487 > Project: Hive > Issue Type: Improvement > Components: SQL >Affects Versions: 0.13.1 >Reporter: N Campbell >Priority: Critical > > While documented > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Union > an application should be able to perform a union query where the projections > are union compatible which does not include the projected column names being > identical which Hive imposes vs ISO-SQL 20xx. > i.e > rejected > select c1 from t1 union all select c2 from t2 > Schema of both sides of union should match. _u1-subquery2 > accepted > select c1 from t1 union all select c2 c1 from t2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15646) Column level lineage is not available for table Views
[ https://issues.apache.org/jira/browse/HIVE-15646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-15646: --- Status: Patch Available (was: Open) > Column level lineage is not available for table Views > - > > Key: HIVE-15646 > URL: https://issues.apache.org/jira/browse/HIVE-15646 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-15646.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15646) Column level lineage is not available for table Views
[ https://issues.apache.org/jira/browse/HIVE-15646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-15646: --- Status: Open (was: Patch Available) > Column level lineage is not available for table Views > - > > Key: HIVE-15646 > URL: https://issues.apache.org/jira/browse/HIVE-15646 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-15646.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15160) Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-15160:
---
Status: Open (was: Patch Available)
> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15160.01.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison,
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
> from catalog_sales
> ,item
> ,date_dim
> where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
> and d_date between cast('2001-01-12' as date)
> and (cast('2001-01-12' as date) + 30 days)
> group by i_item_id
> ,i_item_desc
> ,i_category
> ,i_class
> ,i_current_price
> order by i_category
> ,i_class
> ,i_item_id
> ,i_item_desc
> ,revenueratio
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15578) Simplify IdentifiersParser
[ https://issues.apache.org/jira/browse/HIVE-15578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827160#comment-15827160 ] Pengcheng Xiong commented on HIVE-15578: [~ashutoshc], could u please review? The failed ones are due to legal golden file updates. > Simplify IdentifiersParser > -- > > Key: HIVE-15578 > URL: https://issues.apache.org/jira/browse/HIVE-15578 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-15578.01.patch, HIVE-15578.02.patch > > > before: 1.72M LOC in IdentifiersParser, after: 1.41M -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15160) Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-15160:
---
Status: Patch Available (was: Open)
> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15160.01.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison,
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
> from catalog_sales
> ,item
> ,date_dim
> where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
> and d_date between cast('2001-01-12' as date)
> and (cast('2001-01-12' as date) + 30 days)
> group by i_item_id
> ,i_item_desc
> ,i_category
> ,i_class
> ,i_current_price
> order by i_category
> ,i_class
> ,i_item_id
> ,i_item_desc
> ,revenueratio
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15652) Optimize(reduce) the number of alter calls made to fix repl.last.id
[ https://issues.apache.org/jira/browse/HIVE-15652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-15652: Issue Type: Sub-task (was: Bug) Parent: HIVE-14841 > Optimize(reduce) the number of alter calls made to fix repl.last.id > --- > > Key: HIVE-15652 > URL: https://issues.apache.org/jira/browse/HIVE-15652 > Project: Hive > Issue Type: Sub-task >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-15652.patch > > > Per code review from HIVE-15534, we might be doing alters to parent objects > to set repl.last.id when it is not necessary, since some future event might > make this alter redundant. > There are 3 cases where this might happen: > a) After a CREATE_TABLE event - any prior reference to that table does not > need an ALTER, since CREATE_TABLE will have a repl.last.id come with it. > b) After a DROP_TABLE event - any prior reference to that table is > irrelevant, and thus, no alter is needed. > c) After an ALTER_TABLE event, since that dump will itself do a metadata > update that will get the latest repl.last.id along with this event. > In each of these cases, we can remove the alter call needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15652) Optimize(reduce) the number of alter calls made to fix repl.last.id
[ https://issues.apache.org/jira/browse/HIVE-15652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-15652: Status: Patch Available (was: Open) > Optimize(reduce) the number of alter calls made to fix repl.last.id > --- > > Key: HIVE-15652 > URL: https://issues.apache.org/jira/browse/HIVE-15652 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-15652.patch > > > Per code review from HIVE-15534, we might be doing alters to parent objects > to set repl.last.id when it is not necessary, since some future event might > make this alter redundant. > There are 3 cases where this might happen: > a) After a CREATE_TABLE event - any prior reference to that table does not > need an ALTER, since CREATE_TABLE will have a repl.last.id come with it. > b) After a DROP_TABLE event - any prior reference to that table is > irrelevant, and thus, no alter is needed. > c) After an ALTER_TABLE event, since that dump will itself do a metadata > update that will get the latest repl.last.id along with this event. > In each of these cases, we can remove the alter call needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15531) Hive breaks Hadoop commons logging with log4j2
[ https://issues.apache.org/jira/browse/HIVE-15531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-15531: - Resolution: Won't Fix Status: Resolved (was: Patch Available) Closing it for now as hive enforces the classpath ordering which will avoid the issue. Will revisit later if we run into issues. > Hive breaks Hadoop commons logging with log4j2 > -- > > Key: HIVE-15531 > URL: https://issues.apache.org/jira/browse/HIVE-15531 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Dhiraj Kumar >Assignee: Dhiraj Kumar >Priority: Minor > Attachments: HIVE-15531.patch > > > Hadoop (2.7), which is using Commons-logging is not compatible with log4j2 > without bridge. > The bridge is missing in Hive. > This leads to a problem whereby commons-logging initialises a log4j (1.2) > version Logger, does not configure it properly since configuration for it is > missing and sends logging output to stdout (the default). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15646) Column level lineage is not available for table Views
[ https://issues.apache.org/jira/browse/HIVE-15646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-15646: --- Attachment: HIVE-15646.01.patch > Column level lineage is not available for table Views > - > > Key: HIVE-15646 > URL: https://issues.apache.org/jira/browse/HIVE-15646 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-15646.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15646) Column level lineage is not available for table Views
[ https://issues.apache.org/jira/browse/HIVE-15646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-15646: --- Status: Open (was: Patch Available) > Column level lineage is not available for table Views > - > > Key: HIVE-15646 > URL: https://issues.apache.org/jira/browse/HIVE-15646 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-15646.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15646) Column level lineage is not available for table Views
[ https://issues.apache.org/jira/browse/HIVE-15646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-15646: --- Attachment: (was: HIVE-15646.01.patch) > Column level lineage is not available for table Views > - > > Key: HIVE-15646 > URL: https://issues.apache.org/jira/browse/HIVE-15646 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-15646.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15646) Column level lineage is not available for table Views
[ https://issues.apache.org/jira/browse/HIVE-15646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-15646: --- Status: Patch Available (was: Open) > Column level lineage is not available for table Views > - > > Key: HIVE-15646 > URL: https://issues.apache.org/jira/browse/HIVE-15646 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-15646.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15160) Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-15160:
---
Status: Patch Available (was: Open)
> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15160.01.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison,
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
> from catalog_sales
> ,item
> ,date_dim
> where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
> and d_date between cast('2001-01-12' as date)
> and (cast('2001-01-12' as date) + 30 days)
> group by i_item_id
> ,i_item_desc
> ,i_category
> ,i_class
> ,i_current_price
> order by i_category
> ,i_class
> ,i_item_id
> ,i_item_desc
> ,revenueratio
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15160) Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-15160:
---
Attachment: HIVE-15160.01.patch
> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15160.01.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison,
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
> from catalog_sales
> ,item
> ,date_dim
> where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
> and d_date between cast('2001-01-12' as date)
> and (cast('2001-01-12' as date) + 30 days)
> group by i_item_id
> ,i_item_desc
> ,i_category
> ,i_class
> ,i_current_price
> order by i_category
> ,i_class
> ,i_item_id
> ,i_item_desc
> ,revenueratio
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15160) Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-15160:
---
Attachment: (was: HIVE-15160.01.patch)
> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15160.01.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison,
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
> from catalog_sales
> ,item
> ,date_dim
> where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
> and d_date between cast('2001-01-12' as date)
> and (cast('2001-01-12' as date) + 30 days)
> group by i_item_id
> ,i_item_desc
> ,i_category
> ,i_class
> ,i_current_price
> order by i_category
> ,i_class
> ,i_item_id
> ,i_item_desc
> ,revenueratio
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15160) Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-15160:
---
Status: Open (was: Patch Available)
> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15160.01.patch
>
>
> If a grouping key hasn't been selected, Hive complains. For comparison,
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
> from catalog_sales
> ,item
> ,date_dim
> where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
> and d_date between cast('2001-01-12' as date)
> and (cast('2001-01-12' as date) + 30 days)
> group by i_item_id
> ,i_item_desc
> ,i_category
> ,i_class
> ,i_current_price
> order by i_category
> ,i_class
> ,i_item_id
> ,i_item_desc
> ,revenueratio
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15626) beeline exits on ctrl-c instead of canceling the query
[ https://issues.apache.org/jira/browse/HIVE-15626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827218#comment-15827218 ] Vihang Karajgaonkar commented on HIVE-15626: Hi [~sershe] .. Can you give an example? I am seeing it happening in the master branch as well. Is this issue specific to 1.2.1 or master too? Do you know a branch where this is not happening? > beeline exits on ctrl-c instead of canceling the query > -- > > Key: HIVE-15626 > URL: https://issues.apache.org/jira/browse/HIVE-15626 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Sergey Shelukhin >Assignee: Vihang Karajgaonkar > > I am seeing this in 1.2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15621) Use Hive's own JvmPauseMonitor instead of Hadoop's in LLAP
[
https://issues.apache.org/jira/browse/HIVE-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827226#comment-15827226
]
Hive QA commented on HIVE-15621:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12847941/HIVE-15621.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 372 failed/errored test(s), 10819 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=234)
org.apache.hadoop.hive.accumulo.TestAccumuloConnectionParameters.testMissingPassword
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloConnectionParameters.testMissingUserName
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testDropTableWithoutDeleteLeavesTableIntact
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testEmptyIteratorPushdownValue
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testExternalNonExistentTableFails
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testMissingColumnMappingFails
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testNonBooleanIteratorPushdownValue
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testNonExternalExistentTable
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testNonNullLocation
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testPreCreateTable
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testRollbackCreateTableDeletesExistentTable
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testRollbackCreateTableDoesntDeleteExternalExistentTable
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testRollbackCreateTableOnNonExistentTable
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testTableJobPropertiesCallsInputAndOutputMethods
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testTablePropertiesPassedToInputJobProperties
(batchId=165)
org.apache.hadoop.hive.accumulo.TestAccumuloStorageHandler.testTablePropertiesPassedToOutputJobProperties
(batchId=165)
org.apache.hadoop.hive.accumulo.TestHiveAccumuloHelper.testISEIsPropagated
(batchId=165)
org.apache.hadoop.hive.accumulo.TestHiveAccumuloHelper.testISEIsPropagatedWithReflection
(batchId=165)
org.apache.hadoop.hive.accumulo.TestHiveAccumuloHelper.testTokenMerge
(batchId=165)
org.apache.hadoop.hive.accumulo.TestHiveAccumuloHelper.testTokenToConfFromUser
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableInputFormat.testConfigureAccumuloInputFormat
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableInputFormat.testConfigureAccumuloInputFormatWithAuthorizations
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableInputFormat.testConfigureAccumuloInputFormatWithEmptyColumns
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableInputFormat.testConfigureAccumuloInputFormatWithIterators
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableInputFormat.testConfigureMockAccumuloInputFormat
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableInputFormat.testIteratorNotInSplitsCompensation
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableOutputFormat.testBasicConfiguration
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableOutputFormat.testMockInstance
(batchId=165)
org.apache.hadoop.hive.accumulo.mr.TestHiveAccumuloTableOutputFormat.testSaslConfiguration
(batchId=165)
org.apache.hadoop.hive.accumulo.predicate.TestAccumuloPredicateHandler.testEmptyListRangeGeneratorOutput
(batchId=165)
org.apache.hadoop.hive.accumulo.predicate.TestAccumuloPredicateHandler.testManyRangesGeneratorOutput
(batchId=165)
org.apache.hadoop.hive.accumulo.predicate.TestAccumuloPredicateHandler.testNullRangeGeneratorOutput
(batchId=165)
org.apache.hadoop.hive.accumulo.predicate.TestAccumuloPredicateHandler.testSingleRangeGeneratorOutput
(batchId=165)
org.apache.hadoop.hive.accumulo.serde.TestAccumuloRowSerializer.testBufferResetBeforeUse
(batchId=165)
org.apache.hadoop.hive.cli.TestCliDriverMethods.testThatCliDriverPrintsHeaderForCommandsWithSchema
(batchId=166)
org.apache.hadoop.hive.cli.TestCliDriverMethods.testThatCliDriverPrintsNoHeaderForCommandsWithNoSchema
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_varchar_simple]
(batchId=151)
org.apache.hadoop.hive.common
[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset
[
https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-15519:
--
Attachment: (was: HIVE-15519.6.patch)
> BitSet not computed properly for ColumnBuffer subset
>
>
> Key: HIVE-15519
> URL: https://issues.apache.org/jira/browse/HIVE-15519
> Project: Hive
> Issue Type: Bug
> Components: Hive, JDBC
>Reporter: Bharat Viswanadham
>Assignee: Rui Li
>Priority: Critical
> Attachments: data_type_test(1).txt, HIVE-15519.1.patch,
> HIVE-15519.2.patch, HIVE-15519.3.patch, HIVE-15519.4.patch,
> HIVE-15519.5-branch-1.patch, HIVE-15519.6.patch
>
>
> Hive decimal type column precision is returning as zero, even though column
> has precision set.
> Example: col67 decimal(18,2) scale is returning as zero for that column.
> Tried with below program.
> {code}
>System.out.println("Opening connection");
> Class.forName("org.apache.hive.jdbc.HiveDriver");
>Connection con =
> DriverManager.getConnection("jdbc:hive2://x.x.x.x:1/default");
> DatabaseMetaData dbMeta = con.getMetaData();
>ResultSet rs = dbMeta.getColumns(null, "DEFAULT", "data_type_test",null);
> while (rs.next()) {
> if (rs.getString("COLUMN_NAME").equalsIgnoreCase("col48") ||
> rs.getString("COLUMN_NAME").equalsIgnoreCase("col67") ||
> rs.getString("COLUMN_NAME").equalsIgnoreCase("col68") ||
> rs.getString("COLUMN_NAME").equalsIgnoreCase("col122")){
> System.out.println(rs.getString("COLUMN_NAME") + "\t" +
> rs.getString("COLUMN_SIZE") + "\t" + rs.getInt("DECIMAL_DIGITS"));
> }
>}
>rs.close();
>con.close();
> } catch (Exception e) {
>e.printStackTrace();
>;
> }
> {code}
> Default fetch size is 50. if any column no is under 50 with decimal type,
> precision is returning properly, when the column no is greater than 50, scale
> is returning as zero.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15519) BitSet not computed properly for ColumnBuffer subset
[
https://issues.apache.org/jira/browse/HIVE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-15519:
--
Attachment: HIVE-15519.6.patch
> BitSet not computed properly for ColumnBuffer subset
>
>
> Key: HIVE-15519
> URL: https://issues.apache.org/jira/browse/HIVE-15519
> Project: Hive
> Issue Type: Bug
> Components: Hive, JDBC
>Reporter: Bharat Viswanadham
>Assignee: Rui Li
>Priority: Critical
> Attachments: data_type_test(1).txt, HIVE-15519.1.patch,
> HIVE-15519.2.patch, HIVE-15519.3.patch, HIVE-15519.4.patch,
> HIVE-15519.5-branch-1.patch, HIVE-15519.6.patch
>
>
> Hive decimal type column precision is returning as zero, even though column
> has precision set.
> Example: col67 decimal(18,2) scale is returning as zero for that column.
> Tried with below program.
> {code}
>System.out.println("Opening connection");
> Class.forName("org.apache.hive.jdbc.HiveDriver");
>Connection con =
> DriverManager.getConnection("jdbc:hive2://x.x.x.x:1/default");
> DatabaseMetaData dbMeta = con.getMetaData();
>ResultSet rs = dbMeta.getColumns(null, "DEFAULT", "data_type_test",null);
> while (rs.next()) {
> if (rs.getString("COLUMN_NAME").equalsIgnoreCase("col48") ||
> rs.getString("COLUMN_NAME").equalsIgnoreCase("col67") ||
> rs.getString("COLUMN_NAME").equalsIgnoreCase("col68") ||
> rs.getString("COLUMN_NAME").equalsIgnoreCase("col122")){
> System.out.println(rs.getString("COLUMN_NAME") + "\t" +
> rs.getString("COLUMN_SIZE") + "\t" + rs.getInt("DECIMAL_DIGITS"));
> }
>}
>rs.close();
>con.close();
> } catch (Exception e) {
>e.printStackTrace();
>;
> }
> {code}
> Default fetch size is 50. if any column no is under 50 with decimal type,
> precision is returning properly, when the column no is greater than 50, scale
> is returning as zero.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15653) Some ALTER TABLE commands drop table stats
[
https://issues.apache.org/jira/browse/HIVE-15653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827334#comment-15827334
]
Alexander Behm commented on HIVE-15653:
---
Note that this problem seems to be specific to unpartitioned tables.
Partitioned tables work ok as far as I can tell.
> Some ALTER TABLE commands drop table stats
> --
>
> Key: HIVE-15653
> URL: https://issues.apache.org/jira/browse/HIVE-15653
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.1.0
>Reporter: Alexander Behm
>Priority: Critical
>
> Some ALTER TABLE commands drop the table stats. That may make sense for some
> ALTER TABLE operations, but certainly not for others. Personally, I I think
> ALTER TABLE should only change what was requested by the user without any
> side effects that may be unclear to users. In particular, collecting stats
> can be an expensive operation so it's rather inconvenient for users if they
> get wiped accidentally.
> Repro:
> {code}
> create table t (i int);
> insert into t values(1);
> analyze table t compute statistics;
> alter table t set tblproperties('test'='test');
> hive> describe formatted t;
> OK
> # col_namedata_type comment
>
> i int
>
> # Detailed Table Information
> Database: default
> Owner:abehm
> CreateTime: Tue Jan 17 18:13:34 PST 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Retention:0
> Location: hdfs://localhost:20500/test-warehouse/t
> Table Type: MANAGED_TABLE
> Table Parameters:
> COLUMN_STATS_ACCURATE false
> last_modified_byabehm
> last_modified_time 1484705748
> numFiles1
> numRows -1
> rawDataSize -1
> testtest
> totalSize 2
> transient_lastDdlTime 1484705748
>
> # Storage Information
> SerDe Library:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
>
> InputFormat: org.apache.hadoop.mapred.TextInputFormat
> OutputFormat:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> Compressed: No
> Num Buckets: -1
> Bucket Columns: []
> Sort Columns: []
> Storage Desc Params:
> serialization.format1
> Time taken: 0.169 seconds, Fetched: 34 row(s)
> {code}
> The same behavior can be observed with several other ALTER TABLE commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15591) Hive can not use "," in quoted column name
[ https://issues.apache.org/jira/browse/HIVE-15591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-15591: --- Attachment: HIVE-15591.01.patch > Hive can not use "," in quoted column name > -- > > Key: HIVE-15591 > URL: https://issues.apache.org/jira/browse/HIVE-15591 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-15591.01.patch > > > As reported by [~cartershanklin] > hive> create table test (`x,y` int); > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: > MetaException(message:org.apache.hadoop.hive.serde2.SerDeException > org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe: columns has 2 elements > while columns.types has 1 elements!) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15591) Hive can not use "," in quoted column name
[ https://issues.apache.org/jira/browse/HIVE-15591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-15591: --- Status: Patch Available (was: Open) > Hive can not use "," in quoted column name > -- > > Key: HIVE-15591 > URL: https://issues.apache.org/jira/browse/HIVE-15591 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-15591.01.patch > > > As reported by [~cartershanklin] > hive> create table test (`x,y` int); > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: > MetaException(message:org.apache.hadoop.hive.serde2.SerDeException > org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe: columns has 2 elements > while columns.types has 1 elements!) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
