[jira] [Created] (HIVE-13141) Hive on Spark over HBase should accept parameters starting with "zookeeper.znode"

2016-02-23 Thread Nemon Lou (JIRA) 
Nemon Lou created HIVE-13141:


 Summary: Hive on Spark over HBase should accept parameters 
starting with "zookeeper.znode"
 Key: HIVE-13141
 URL: https://issues.apache.org/jira/browse/HIVE-13141
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.0.0, 1.2.0
Reporter: Nemon Lou
Assignee: Nemon Lou
Priority: Minor


HBase related paramters has been added by HIVE-12708.
Following the same way,parameters starting with "zookeeper.znode" should be add 
too,which are also HBase related paramters .
Refering to http://blog.cloudera.com/blog/2013/10/what-are-hbase-znodes/

I have seen a failure with Hive on Spark over HBase  due to customize 
zookeeper.znode.parent.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13140) Wire the client to submit execution fragments

2016-02-23 Thread Siddharth Seth (JIRA) 
Siddharth Seth created HIVE-13140:
-

 Summary: Wire the client to submit execution fragments
 Key: HIVE-13140
 URL: https://issues.apache.org/jira/browse/HIVE-13140
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13139) Unfold TOK_ALLCOLREF at QB stage

2016-02-23 Thread Pengcheng Xiong (JIRA) 
Pengcheng Xiong created HIVE-13139:
--

 Summary: Unfold TOK_ALLCOLREF at QB stage
 Key: HIVE-13139
 URL: https://issues.apache.org/jira/browse/HIVE-13139
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


TOK_ALLCOLREF is now unfolded to real column names at genPlan stage. This patch 
tries to unfold it at QB stage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 43834: Support view column authorization

2016-02-23 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43834/
---

(Updated Feb. 24, 2016, 7:44 a.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-13095


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 10bd97b 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPruner.java c353e3e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 
78bce23 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java
 18145ae 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 809affb 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 642c227 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java f04b493 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 8a06582 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java fc555ca 
  ql/src/test/queries/clientnegative/authorization_view_1.q PRE-CREATION 
  ql/src/test/queries/clientnegative/authorization_view_2.q PRE-CREATION 
  ql/src/test/queries/clientnegative/authorization_view_3.q PRE-CREATION 
  ql/src/test/queries/clientnegative/authorization_view_4.q PRE-CREATION 
  ql/src/test/queries/clientnegative/authorization_view_disable_cbo_1.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/authorization_view_disable_cbo_2.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/authorization_view_disable_cbo_3.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/authorization_view_disable_cbo_4.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/authorization_view_1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/authorization_view_disable_cbo_1.q 
PRE-CREATION 
  ql/src/test/results/clientnegative/authorization_view_1.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/authorization_view_2.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/authorization_view_3.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/authorization_view_4.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/authorization_view_disable_cbo_1.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/authorization_view_disable_cbo_2.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/authorization_view_disable_cbo_3.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/authorization_view_disable_cbo_4.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/authorization_view_1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/authorization_view_disable_cbo_1.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/43834/diff/


Testing
---


Thanks,

pengcheng xiong

[jira] [Created] (HIVE-13138) Add an initial client to communicate with the daemons

2016-02-23 Thread Siddharth Seth (JIRA) 
Siddharth Seth created HIVE-13138:
-

 Summary: Add an initial client to communicate with the daemons
 Key: HIVE-13138
 URL: https://issues.apache.org/jira/browse/HIVE-13138
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 43408: HIVE-13034 : Add jdeb plugin to build debian

2016-02-23 Thread Arshad Matin 


> On Feb. 11, 2016, 10:31 a.m., Amareshwari Sriramadasu wrote:
> > packaging/pom.xml, line 92
> > 
> >
> > Can we move to latest verison 1.5 ?
> 
> Arshad Matin wrote:
> sure

For some reason it is not working with 1.5, updated the version to 1.0.1


- Arshad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43408/#review118848
---


On Feb. 24, 2016, 7:07 a.m., Arshad Matin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43408/
> ---
> 
> (Updated Feb. 24, 2016, 7:07 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-13034
> https://issues.apache.org/jira/browse/HIVE-13034
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13034 : Add jdeb plugin to build debian
> 
> 
> Diffs
> -
> 
>   packaging/pom.xml e7feb62 
>   packaging/src/deb/control/control PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/43408/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Arshad Matin
> 
>

Re: Review Request 43408: HIVE-13034 : Add jdeb plugin to build debian

2016-02-23 Thread Arshad Matin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43408/
---

(Updated Feb. 24, 2016, 7:07 a.m.)


Review request for hive.


Bugs: HIVE-13034
https://issues.apache.org/jira/browse/HIVE-13034


Repository: hive-git


Description
---

HIVE-13034 : Add jdeb plugin to build debian


Diffs (updated)
-

  packaging/pom.xml e7feb62 
  packaging/src/deb/control/control PRE-CREATION 

Diff: https://reviews.apache.org/r/43408/diff/


Testing
---


Thanks,

Arshad Matin

[jira] [Created] (HIVE-13137) Hive parse_url(url,'HOST') returns null values

2016-02-23 Thread dhruv kapatel (JIRA) 
dhruv kapatel created HIVE-13137:


 Summary: Hive parse_url(url,'HOST') returns null values
 Key: HIVE-13137
 URL: https://issues.apache.org/jira/browse/HIVE-13137
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.0.1
 Environment: ubuntu 12.04
Reporter: dhruv kapatel


I've table with url stored in it. I want to extract host name from urls.
I've used this query
SELECT parse_url(url,'HOST') AS host FROM table;
It returns null for all rows.
SELECT parse_url('http://www.google.co.in','HOST') returns www.google.co.in.

 . 
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 43408: HIVE-13034 : Add jdeb plugin to build debian

2016-02-23 Thread Arshad Matin 


> On Feb. 10, 2016, 10:57 a.m., Amareshwari Sriramadasu wrote:
> > packaging/pom.xml, line 108
> > 
> >
> > Directory should /usr/local/hive/bin/ instead of top level hive/

It will have bin directory inside "/usr/local/hive" after installation.
Example : 
arshad.matin@:/usr/local/hive$ ls
bin  conf  examples  hcatalog  lib  LICENSE  NOTICE  README.txt  
RELEASE_NOTES.txt  scripts


> On Feb. 10, 2016, 10:57 a.m., Amareshwari Sriramadasu wrote:
> > packaging/src/deb/control/control, line 20
> > 
> >
> > Should we make it java7 ?

sure


> On Feb. 10, 2016, 10:57 a.m., Amareshwari Sriramadasu wrote:
> > packaging/src/deb/control/control, line 21
> > 
> >
> > Should be dev@hive.apache.org

will make the changes


- Arshad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43408/#review118623
---


On Feb. 22, 2016, 9:11 a.m., Arshad Matin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43408/
> ---
> 
> (Updated Feb. 22, 2016, 9:11 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-13034
> https://issues.apache.org/jira/browse/HIVE-13034
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13034 : Add jdeb plugin to build debian
> 
> 
> Diffs
> -
> 
>   packaging/pom.xml e7feb62 
>   packaging/src/deb/control/control PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/43408/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Arshad Matin
> 
>

Re: Review Request 43408: HIVE-13034 : Add jdeb plugin to build debian

2016-02-23 Thread Arshad Matin 


> On Feb. 11, 2016, 10:31 a.m., Amareshwari Sriramadasu wrote:
> > packaging/pom.xml, line 92
> > 
> >
> > Can we move to latest verison 1.5 ?

sure


- Arshad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43408/#review118848
---


On Feb. 22, 2016, 9:11 a.m., Arshad Matin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43408/
> ---
> 
> (Updated Feb. 22, 2016, 9:11 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-13034
> https://issues.apache.org/jira/browse/HIVE-13034
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13034 : Add jdeb plugin to build debian
> 
> 
> Diffs
> -
> 
>   packaging/pom.xml e7feb62 
>   packaging/src/deb/control/control PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/43408/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Arshad Matin
> 
>

[jira] [Created] (HIVE-13136) Create index failed because index table not found.

2016-02-23 Thread Meng, Yongjian (JIRA) 
Meng, Yongjian created HIVE-13136:
-

 Summary: Create index failed because index table not found.
 Key: HIVE-13136
 URL: https://issues.apache.org/jira/browse/HIVE-13136
 Project: Hive
  Issue Type: Bug
  Components: Beeline, hpl/sql
Affects Versions: 1.2.1
Reporter: Meng, Yongjian
Assignee: Meng, Yongjian


When I execute 
CREATE INDEX y_key1_index ON TABLE y(key1) AS 
'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler' WITH DEFERRED 
REBUILD;
It always display an Error: Error while processing statement: FAILED: Execution 
Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Table not 
found default__y_y_key1_index__ (state=08S01,code=1)
BUT in my hdfs, there exists the index table, but hive cannot load it when I 
show tables.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13135) LLAP: HTTPS Webservices needs trusted keystore configs

2016-02-23 Thread Gopal V (JIRA) 
Gopal V created HIVE-13135:
--

 Summary: LLAP: HTTPS Webservices needs trusted keystore configs
 Key: HIVE-13135
 URL: https://issues.apache.org/jira/browse/HIVE-13135
 Project: Hive
  Issue Type: Bug
  Components: llap
Affects Versions: 2.1.0
Reporter: Gopal V
Assignee: Gopal V
 Fix For: 2.1.0


ssl-server.xml is not picked up internally by the hive-common HttpServer impl, 
unlike the default configs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13134) JDBC: JDBC Standalone should not be in the lib dir by default

2016-02-23 Thread Gopal V (JIRA) 
Gopal V created HIVE-13134:
--

 Summary: JDBC: JDBC Standalone should not be in the lib dir by 
default
 Key: HIVE-13134
 URL: https://issues.apache.org/jira/browse/HIVE-13134
 Project: Hive
  Issue Type: Bug
  Components: JDBC
Reporter: Gopal V
Assignee: Vaibhav Gumashta






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13133) Create initial InputFormat + record readers/writers

2016-02-23 Thread Gunther Hagleitner (JIRA) 
Gunther Hagleitner created HIVE-13133:
-

 Summary: Create initial InputFormat + record readers/writers
 Key: HIVE-13133
 URL: https://issues.apache.org/jira/browse/HIVE-13133
 Project: Hive
  Issue Type: Sub-task
Reporter: Gunther Hagleitner






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13132) Hive should lazily load and cache metastore (permanent) functions

2016-02-23 Thread Anthony Hsu (JIRA) 
Anthony Hsu created HIVE-13132:
--

 Summary: Hive should lazily load and cache metastore (permanent) 
functions
 Key: HIVE-13132
 URL: https://issues.apache.org/jira/browse/HIVE-13132
 Project: Hive
  Issue Type: Improvement
Affects Versions: 0.13.1
Reporter: Anthony Hsu
Assignee: Anthony Hsu


In Hive 0.13.1, we have noticed that as the number of databases increases, the 
start-up time of the Hive interactive shell increases. This is because during 
start-up, all databases are iterated over to fetch the permanent functions to 
display in the {{SHOW FUNCTIONS}} output.

{noformat:title=FunctionRegistry.java}
  private static Set getFunctionNames(boolean searchMetastore) {
Set functionNames = mFunctions.keySet();
if (searchMetastore) {
  functionNames = new HashSet(functionNames);
  try {
Hive db = getHive();
List dbNames = db.getAllDatabases();

for (String dbName : dbNames) {
  List funcNames = db.getFunctions(dbName, "*");
  for (String funcName : funcNames) {
functionNames.add(FunctionUtils.qualifyFunctionName(funcName, 
dbName));
  }
}
  } catch (Exception e) {
LOG.error(e);
// Continue on, we can still return the functions we've gotten to this 
point.
  }
}
return functionNames;
  }
{noformat}

Instead of eagerly loading all metastore functions, we should only load them 
the first time {{SHOW FUNCTIONS}} is invoked. We should also cache the results.

Note that this issue may have been fixed by HIVE-2573, though I haven't 
verified this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13131) TezWork queryName can be null after HIVE-12523

2016-02-23 Thread Jason Dere (JIRA) 
Jason Dere created HIVE-13131:
-

 Summary: TezWork queryName can be null after HIVE-12523
 Key: HIVE-13131
 URL: https://issues.apache.org/jira/browse/HIVE-13131
 Project: Hive
  Issue Type: Bug
  Components: Tez
Reporter: Jason Dere
Assignee: Vikram Dixit K


Looks like after HIVE-12523, the queryName field can be null, either if the 
conf passed in is null, or if the conf does not contain the necessary settings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13130) API calls for retrieving primary keys and foreign keys information

2016-02-23 Thread Hari Sankar Sivarama Subramaniyan (JIRA) 
Hari Sankar Sivarama Subramaniyan created HIVE-13130:


 Summary:  API calls for retrieving primary keys and foreign keys 
information
 Key: HIVE-13130
 URL: https://issues.apache.org/jira/browse/HIVE-13130
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan


ODBC exposes the SQLPrimaryKeys and SQLForeignKeys API calls and JDBC exposes 
getPrimaryKeys and getCrossReference API calls. We need to provide these 
interfaces as part of PK/FK implementation in Hive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 43921: HIVE-13120 propagate doAs when generating ORC splits

2016-02-23 Thread Gopal V 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43921/#review120418
---




ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java (line 731)


Earlier, the FileSystem object used to handle the UGI issues - did we start 
passing NULL here for some reason?


- Gopal V


On Feb. 24, 2016, 12:12 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43921/
> ---
> 
> (Updated Feb. 24, 2016, 12:12 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java f36f707 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> 9b1d7ae 
> 
> Diff: https://reviews.apache.org/r/43921/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Re: Review Request 43921: HIVE-13120 propagate doAs when generating ORC splits

2016-02-23 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43921/#review120417
---




ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java (line 545)


not needed


- Sergey Shelukhin


On Feb. 24, 2016, 12:12 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43921/
> ---
> 
> (Updated Feb. 24, 2016, 12:12 a.m.)
> 
> 
> Review request for hive and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java f36f707 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> 9b1d7ae 
> 
> Diff: https://reviews.apache.org/r/43921/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>

Review Request 43921: HIVE-13120 propagate doAs when generating ORC splits

2016-02-23 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43921/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java f36f707 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
9b1d7ae 

Diff: https://reviews.apache.org/r/43921/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-13129) CliService leaks HMS connection

2016-02-23 Thread Aihua Xu (JIRA) 
Aihua Xu created HIVE-13129:
---

 Summary: CliService leaks HMS connection
 Key: HIVE-13129
 URL: https://issues.apache.org/jira/browse/HIVE-13129
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 2.1.0
Reporter: Aihua Xu
Assignee: Aihua Xu


HIVE-12790 fixes the HMS connection leaking. But seems there is one more 
connection from CLIService.

The init() function in CLIService will get info from DB but we never close the 
HMS connection. 

{noformat}
// creates connection to HMS and thus *must* occur after kerberos login 
above
try {
  applyAuthorizationConfigPolicy(hiveConf);
} catch (Exception e) {
  throw new RuntimeException("Error applying authorization policy on hive 
configuration: "
  + e.getMessage(), e);
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 43834: Support view column authorization

2016-02-23 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43834/#review120371
---



It will also be good to add true junit tests for this. You may look at 
TestHiveAuthorizerCheckInvocation for pattern.


ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java
 (line 403)


Instead of repeating all the logic here, shall we call super() here?



ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java (line 50)


better name: viewAliasToViewSchema?



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (lines 10439 
- 10449)


Do we really need to invoke CP here? When CP gets invoked later on in 
optimize phase, it will populate column access info correctly, right?



ql/src/test/queries/clientnegative/authorization_view_1.q (line 12)


Add:
grant select(key) on v to user hive_test_user;

Test should still fail, since hive_test_user doesnt have select privs for 
value.



ql/src/test/queries/clientnegative/authorization_view_2.q (line 11)


Change user to hive_test_user for both statements.



ql/src/test/queries/clientnegative/authorization_view_disable_cbo_1.q (line 12)


Similar comment as above.



ql/src/test/queries/clientnegative/authorization_view_disable_cbo_3.q (line 11)


User name: hive_test_user


- Ashutosh Chauhan


On Feb. 22, 2016, 3:24 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43834/
> ---
> 
> (Updated Feb. 22, 2016, 3:24 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13095
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 10bd97b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java 
> 78bce23 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java
>  18145ae 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 809affb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 642c227 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java f04b493 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 8a06582 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java fc555ca 
>   ql/src/test/queries/clientnegative/authorization_view_1.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/authorization_view_2.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/authorization_view_disable_cbo_1.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/authorization_view_disable_cbo_2.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/authorization_view_disable_cbo_3.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/authorization_view_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/authorization_view_disable_cbo_1.q 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/authorization_view_1.q.out PRE-CREATION 
>   ql/src/test/results/clientnegative/authorization_view_2.q.out PRE-CREATION 
>   ql/src/test/results/clientnegative/authorization_view_disable_cbo_1.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/authorization_view_disable_cbo_2.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/authorization_view_disable_cbo_3.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/authorization_view_1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/authorization_view_disable_cbo_1.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/43834/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

[jira] [Created] (HIVE-13128) NullScan fails on a secure setup

2016-02-23 Thread Siddharth Seth (JIRA) 
Siddharth Seth created HIVE-13128:
-

 Summary: NullScan fails on a secure setup
 Key: HIVE-13128
 URL: https://issues.apache.org/jira/browse/HIVE-13128
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.1.0
Reporter: Siddharth Seth
Assignee: Siddharth Seth
Priority: Critical


Nullscan provides uris of the form nullscan://null/ - which are added to the 
list of FileSystems for which Tez should obtain tokens.

{code}
2016-02-19T02:48:04,481 ERROR [main]: exec.Task (TezTask.java:execute(219)) - 
Failed to execute tez graph.
java.lang.IllegalArgumentException: java.net.UnknownHostException: null
  at 
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:406)
 ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
  at 
org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:291)
 ~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
  at 
org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:302) 
~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
  at 
org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:524) 
~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
  at org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:508) 
~[hadoop-common-2.7.1.2.3.5.1-26.jar:?]
  at 
org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:107)
 ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
  at 
org.apache.tez.common.security.TokenCache.obtainTokensForFileSystemsInternal(TokenCache.java:86)
 ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
  at 
org.apache.tez.common.security.TokenCache.obtainTokensForFileSystems(TokenCache.java:76)
 ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
  at 
org.apache.tez.client.TezClientUtils.addFileSystemCredentialsFromURIs(TezClientUtils.java:338)
 ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
  at 
org.apache.tez.client.TezClientUtils.setupDAGCredentials(TezClientUtils.java:369)
 ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
  at 
org.apache.tez.client.TezClientUtils.prepareAndCreateDAGPlan(TezClientUtils.java:704)
 ~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
  at org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:522) 
~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
  at org.apache.tez.client.TezClient.submitDAG(TezClient.java:468) 
~[tez-api-0.8.2.2.3.5.1-26.jar:0.8.2.2.3.5.1-26]
  at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:466) 
~[hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:187) 
[hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:158) 
[hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89) 
[hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858) 
[hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1600) 
[hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1373) 
[hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1196) 
[hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1184) 
[hive-exec-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:228) 
[hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:180) 
[hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:395) 
[hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:331) 
[hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:428) 
[hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:444) 
[hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:744) 
[hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:711) 
[hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:640) 
[hive-cli-2.0.0.2.3.5.1-26.jar:2.0.0.2.3.5.1-26]
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]
  at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_45]
  at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_45]
  at

Re: Review Request 43885: HIVE-13102

2016-02-23 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43885/#review120363
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexUtil.java (line 
199)


Can you add a comment about why we are treating inverted and negated terms 
in same manner. My assumption was we can fold to false only in presence of 
negated terms, not inverted terms.



ql/src/test/results/clientpositive/ppd_udf_col.q.out (line 117)


Seems like we missed opportunity to fold constant here.


- Ashutosh Chauhan


On Feb. 23, 2016, 5:27 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43885/
> ---
> 
> (Updated Feb. 23, 2016, 5:27 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-13102
> https://issues.apache.org/jira/browse/HIVE-13102
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13102
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> 58a7cff97cf8120b7afdb52af6a8f6cd07b408ea 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexUtil.java 
> 3f6dd6a3738bdcb3c4756136965c54ea6f943f53 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinPushTransitivePredicatesRule.java
>  994af976f12f8b59c2379f23a33cf8fc94a2efc5 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
>  8f15ec7e5f075ceee1ca71b9a389837b9a31167e 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdPredicates.java
>  36d0b458d4c33707f8e3a5b823e35edf8c5d09d1 
>   ql/src/test/queries/clientpositive/constprog3.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/infer_join_preds.q PRE-CREATION 
>   ql/src/test/results/clientpositive/constprog3.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/fold_case.q.out 
> 90ea0afa49673910be4b28ce5f96e942b42936b0 
>   ql/src/test/results/clientpositive/infer_join_preds.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/mergejoin.q.out 
> f2c04e2221c1e7164ffb7cfe49f2fe3e0536e321 
>   ql/src/test/results/clientpositive/partition_boolexpr.q.out 
> 5272f33621a45616895f81684e4f620f6b0ec765 
>   ql/src/test/results/clientpositive/ppd_udf_col.q.out 
> 2641f5c89a1a8839e8c3e5bfa5f4b6d756ec47fb 
>   ql/src/test/results/clientpositive/tez/mergejoin.q.out 
> 14d04317d17684877566ae4aad29a862255ba02c 
> 
> Diff: https://reviews.apache.org/r/43885/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

[jira] [Created] (HIVE-13127) Rebase LLAP branch with master

2016-02-23 Thread Jason Dere (JIRA) 
Jason Dere created HIVE-13127:
-

 Summary: Rebase LLAP branch with master
 Key: HIVE-13127
 URL: https://issues.apache.org/jira/browse/HIVE-13127
 Project: Hive
  Issue Type: Sub-task
Reporter: Jason Dere
Assignee: Jason Dere


Update llap branch with the current state of master



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13126) Clean up MapJoinOperator properly to avoid object cache reuse with unintentional states

2016-02-23 Thread Wei Zheng (JIRA) 
Wei Zheng created HIVE-13126:


 Summary: Clean up MapJoinOperator properly to avoid object cache 
reuse with unintentional states
 Key: HIVE-13126
 URL: https://issues.apache.org/jira/browse/HIVE-13126
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.0.0
Reporter: Wei Zheng
Assignee: Wei Zheng


For a given job, one task may reuse other task's object cache (plan cache) such 
as MapJoinOperator. This is fine. But if we have some dirty states left over, 
it may cause issue like wrong results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 43885: HIVE-13102

2016-02-23 Thread Jesús Camacho Rodríguez 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43885/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-13102
https://issues.apache.org/jira/browse/HIVE-13102


Repository: hive-git


Description
---

HIVE-13102


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
58a7cff97cf8120b7afdb52af6a8f6cd07b408ea 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexUtil.java 
3f6dd6a3738bdcb3c4756136965c54ea6f943f53 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinPushTransitivePredicatesRule.java
 994af976f12f8b59c2379f23a33cf8fc94a2efc5 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
 8f15ec7e5f075ceee1ca71b9a389837b9a31167e 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdPredicates.java
 36d0b458d4c33707f8e3a5b823e35edf8c5d09d1 
  ql/src/test/queries/clientpositive/constprog3.q PRE-CREATION 
  ql/src/test/queries/clientpositive/infer_join_preds.q PRE-CREATION 
  ql/src/test/results/clientpositive/constprog3.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/fold_case.q.out 
90ea0afa49673910be4b28ce5f96e942b42936b0 
  ql/src/test/results/clientpositive/infer_join_preds.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/mergejoin.q.out 
f2c04e2221c1e7164ffb7cfe49f2fe3e0536e321 
  ql/src/test/results/clientpositive/partition_boolexpr.q.out 
5272f33621a45616895f81684e4f620f6b0ec765 
  ql/src/test/results/clientpositive/ppd_udf_col.q.out 
2641f5c89a1a8839e8c3e5bfa5f4b6d756ec47fb 
  ql/src/test/results/clientpositive/tez/mergejoin.q.out 
14d04317d17684877566ae4aad29a862255ba02c 

Diff: https://reviews.apache.org/r/43885/diff/


Testing
---


Thanks,

Jesús Camacho Rodríguez

[jira] [Created] (HIVE-13125) Support masking and filtering of rows/columns

2016-02-23 Thread Pengcheng Xiong (JIRA) 
Pengcheng Xiong created HIVE-13125:
--

 Summary: Support masking and filtering of rows/columns
 Key: HIVE-13125
 URL: https://issues.apache.org/jira/browse/HIVE-13125
 Project: Hive
  Issue Type: New Feature
Reporter: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13124) Join external tables with the same location fails for MR

2016-02-23 Thread Sebastian Walz (JIRA) 
Sebastian Walz created HIVE-13124:
-

 Summary: Join external tables with the same location fails for MR
 Key: HIVE-13124
 URL: https://issues.apache.org/jira/browse/HIVE-13124
 Project: Hive
  Issue Type: Bug
  Components: Database/Schema, Hive, Query Processor, 
Serializers/Deserializers
Affects Versions: 1.2.1
 Environment: Hortonworks VM:
Hive 1.2.1
Hadoop 2.7.1.2.3.2.0-2950

Reporter: Sebastian Walz


There seems to be a bug in Hive using Map-Reduce as execution engine when you 
have multiple external tables with the same location and you try to join them.
(in my case the files in the location are XML-files and I created a custom 
SerDe for reading those "non-standard" XML-files.) 
When I am trying to execute a "select ... join " the task will fail with 
this message:

2016-02-23 08:06:35,976 ERROR [main]: exec.Task 
(SessionState.java:printError(960)) - Execution failed with exit status: 2
2016-02-23 08:06:35,977 ERROR [main]: exec.Task 
(SessionState.java:printError(960)) - Obtaining error information
2016-02-23 08:06:35,977 ERROR [main]: exec.Task 
(SessionState.java:printError(960)) - 
Task failed!
Task ID:
  Stage-4


And this is the exception from the log:

2016-02-23 08:06:35,756 ERROR mr.MapredLocalTask 
(MapredLocalTask.java:executeInProcess(355)) - Hive Runtime Error: Map local 
work failed
java.lang.RuntimeException: cannot find field feb45_vehicleinfo_id from 
[0:feb06_event_id, 1:feb01_feedbacksession_id, 2:feb07_eventtype_id, 
3:feb06_event_ts]
at 
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:416)
at 
org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:147)
at 
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at 
org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:77)
at 
org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:147)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:363)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:482)
at 
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:439)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:482)
at 
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:439)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:458)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:364)
at 
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:344)
at 
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:747)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)


If I copy the files to a different directory and point in each table to its own 
directory, than I don't have this problem. But I don't want to copy tons of 
gigabyte 35 times on the cluster as a workaround. Tez also does not have this 
kind of issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)