date:20160518

[jira] [Created] (HIVE-13788) hive msck listpartitions need to make use of directSQL instead of datanucleus

2016-05-18 Thread Rajesh Balamohan (JIRA)

Rajesh Balamohan created HIVE-13788:
---

 Summary: hive msck listpartitions need to make use of directSQL 
instead of datanucleus
 Key: HIVE-13788
 URL: https://issues.apache.org/jira/browse/HIVE-13788
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan
Priority: Minor


Currently, for tables having 1000s of partitions too many DB calls are made via 
datanucleus.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13787) LLAP: bug in recent security patches

2016-05-18 Thread Sergey Shelukhin (JIRA)

Sergey Shelukhin created HIVE-13787:
---

 Summary: LLAP: bug in recent security patches
 Key: HIVE-13787
 URL: https://issues.apache.org/jira/browse/HIVE-13787
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Hive system catalog

2016-05-18 Thread brajmohan saxena

Hi,

Could any body please help me to know how to get the Hive system catalog
tables.
Also I am looking for a catalog table which can show me all the built in
functions list of Hive.

Thanks in advance.

Regards
Braj

Re: Becoming a contributor

2016-05-18 Thread Alan Gates

You’ve been added, so you should now be able to assign JIRAs to yourself.

Alan.

> On May 18, 2016, at 13:38, Vihang Karajgaonkar  wrote:
> 
> Thanks Alan! My JIRA id is vihangk1
> 
> -Vihang
> 
>> On May 18, 2016, at 1:09 PM, Alan Gates  wrote:
>> 
>> Nope, you’re good.  You can ignore the stuff about review board as we 
>> generally only use that now for large and complex patches.  If you have a 
>> JIRA in mind you’d like to work on that no one else has you can assign it to 
>> yourself* and get started.  If someone else is working on it you should 
>> coordinate via comments on that JIRA to see if there’s a way you can help.
>> 
>> Welcome to the team.
>> 
>> Alan.
>> 
>> *You probably can’t assign JIRAs to yourself yet, but if you reply to this 
>> email with your JIRA id I’ll make it so you can.
>> 
>> Alan.
>> 
>>> On May 18, 2016, at 11:42, Vihang Karajgaonkar  wrote:
>>> 
>>> Hello Everyone,
>>> 
>>> I would like to start working on issues reported on Hive in JIRA. I 
>>> followed the steps mentioned in 
>>> https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-BecomingaContributor
>>>  
>>> .
>>>  Is there anything else that I need to do to get added as a contributor?
>>> 
>>> Thanks,
>>> Vihang
>> 
>

Re: Review Request 47554: HIVE-13750

2016-05-18 Thread Ashutosh Chauhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47554/#review133844
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
 (line 517)


This is already called, before a call to procesDynP..
Do we need it again?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
 (line 536)


Can you add comments what are permissible operator sequences here?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
 (line 554)


Can you also add comment why this is valid only for RS added by SPDO and 
not in general?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
 (line 582)


Currently, _buceket_number_ comes in as constant when it really its not. 
Will that break assumption about ignoring constant?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
 (lines 738 - 742)


Should it be other way round? That we first try normal dedup and then 
extended.


- Ashutosh Chauhan


On May 18, 2016, 9:42 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47554/
> ---
> 
> (Updated May 18, 2016, 9:42 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-13750
> https://issues.apache.org/jira/browse/HIVE-13750
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13750
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java
>  010c89ed978296709b052cc7bc80256a27658e2b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
>  733620b84657a21829248afe72ab16ad9692f37e 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 
> d7e404c9946461e20357ed53dd8da468590683c6 
>   ql/src/test/results/clientpositive/dynpart_sort_opt_vectorization.q.out 
> d03bfe422743d9a5a6b85f9a6198e1e27024f129 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization.q.out 
> dec872ab0eef54bd92d5c2bc068e2805cc14e272 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
> 832580325873dee741ba86239ee571873994a808 
>   ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
> a90e3f63b4646cf0ade9785a501ebd1a6b2a3406 
>   ql/src/test/results/clientpositive/tez/dynpart_sort_optimization.q.out 
> 723e8192f2735059005fc3c5c96732a2c4be49c1 
> 
> Diff: https://reviews.apache.org/r/47554/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

Re: Review Request 47419: enable merging of bit vectors for insert into

2016-05-18 Thread pengcheng xiong



> On May 18, 2016, 11:04 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java,
> >  line 102
> > 
> >
> > For static partitions no need to specify part-spec ?

no need


> On May 18, 2016, 11:04 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java,
> >  line 151
> > 
> >
> > Will this trigger new calls to metastore to collect metadata ?

sure, we should cache


> On May 18, 2016, 11:04 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java,
> >  line 157
> > 
> >
> > Might make sense to throw an exception if values().size() > 1

agree


> On May 18, 2016, 11:04 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java, lines 267-277
> > 
> >
> > Wanna move this utility BFS search method in some utils class. Might be 
> > useful for other purposes as well.

agree


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47419/#review133828
---


On May 16, 2016, 6:11 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47419/
> ---
> 
> (Updated May 16, 2016, 6:11 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13566
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 541af57 
>   itests/src/test/resources/testconfiguration.properties c891d40 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java d8ac6ae 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
> d213731 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java
>  9fbbd4c 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 
> 02c5a89 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
> 3b6cbce 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 96ef20d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 3a226e7 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 7162c08 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 4049f40 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java 
> 7433263 
>   ql/src/test/queries/clientpositive/autoColumnStats_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/autoColumnStats_2.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/autoColumnStats_3.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/autoColumnStats_4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/autoColumnStats_5.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/autoColumnStats_6.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/autoColumnStats_7.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/autoColumnStats_8.q PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_3.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_5.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_6.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_7.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_8.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/autoColumnStats_2.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/47419/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 47419: enable merging of bit vectors for insert into

2016-05-18 Thread Ashutosh Chauhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47419/#review133828
---




ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java 
(line 102)


For static partitions no need to specify part-spec ?



ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java 
(line 151)


Will this trigger new calls to metastore to collect metadata ?



ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java 
(line 157)


Might make sense to throw an exception if values().size() > 1



ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java 
(line 215)


Can you add a comment in what case srcType != destType



ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
(line 399)


Better name: rewriteAST() ?



ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java (lines 66 - 67)


Can you please add comments describing purpose of these new data structures?



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (lines 7052 - 
7062)


This will add latency in query compilation because this makes new calls to 
metastore to get table and partition objects. 
An alternative could be to always add CS pipeline and then delete it later 
(if needed) either in FetchOptimizer where ParseContext already have these 
objects or even later in TaskCompiler.



ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java (lines 267 - 277)


Wanna move this utility BFS search method in some utils class. Might be 
useful for other purposes as well.


Thanks for nicely commenting the patch.
Overall looks good, my major concern is around how much compile latency this 
patch may increase?

- Ashutosh Chauhan


On May 16, 2016, 6:11 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47419/
> ---
> 
> (Updated May 16, 2016, 6:11 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13566
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 541af57 
>   itests/src/test/resources/testconfiguration.properties c891d40 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java d8ac6ae 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java 
> d213731 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java
>  9fbbd4c 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java 
> 02c5a89 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java 
> PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
> 3b6cbce 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 96ef20d 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java 3a226e7 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 7162c08 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 4049f40 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBridge.java 
> 7433263 
>   ql/src/test/queries/clientpositive/autoColumnStats_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/autoColumnStats_2.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/autoColumnStats_3.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/autoColumnStats_4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/autoColumnStats_5.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/autoColumnStats_6.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/autoColumnStats_7.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/autoColumnStats_8.q PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_3.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_5.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_6.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/autoColumnStats_7.q.out P

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

2016-05-18 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47563/
---

(Updated May 18, 2016, 11:04 p.m.)


Review request for hive, Jason Dere and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA. Please ignore the first iteration, RB doesn't allow base patches on 
submit


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cbb3a72 
  llap-client/src/java/org/apache/hadoop/hive/llap/SubmitWorkInfo.java 6704294 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
6c2618b 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 f10351b 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClientImpl.java
 PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinator.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java e43b72b 
  llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
349ee14 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
4306c22 
  
llap-server/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinatorImpl.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 2524dc2 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
PRE-CREATION 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 026df3b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
ce69ee6 
  service/src/java/org/apache/hive/service/server/HiveServer2.java d61edf5 

Diff: https://reviews.apache.org/r/47563/diff/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

2016-05-18 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47563/
---

Review request for hive, Jason Dere and Siddharth Seth.


Repository: hive-git


Description
---

see JIRA. Please ignore the first iteration, RB doesn't allow base patches on 
submit


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cbb3a72 
  llap-client/src/java/org/apache/hadoop/hive/llap/SubmitWorkInfo.java 6704294 
  llap-client/src/java/org/apache/hadoop/hive/llap/io/api/LlapProxy.java 
6c2618b 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 f10351b 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClientImpl.java
 PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinator.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 e28eddd 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
465b204 
  llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java e43b72b 
  llap-common/src/test/org/apache/hadoop/hive/llap/tez/TestConverters.java 
349ee14 
  llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java 
4306c22 
  
llap-server/src/java/org/apache/hadoop/hive/llap/coordinator/LlapCoordinatorImpl.java
 PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 2524dc2 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
de817e3 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 b94fc2e 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
 03ee055 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
8abd198 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 eac0e8f 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 74359fa 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
PRE-CREATION 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 279baf1 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
 762 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 a250882 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
 026df3b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
ce69ee6 
  service/src/java/org/apache/hive/service/server/HiveServer2.java d61edf5 

Diff: https://reviews.apache.org/r/47563/diff/


Testing
---


Thanks,

Sergey Shelukhin

Review Request 47554: HIVE-13750

2016-05-18 Thread Jesús Camacho Rodríguez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47554/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-13750
https://issues.apache.org/jira/browse/HIVE-13750


Repository: hive-git


Description
---

HIVE-13750


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java
 010c89ed978296709b052cc7bc80256a27658e2b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
 733620b84657a21829248afe72ab16ad9692f37e 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 
d7e404c9946461e20357ed53dd8da468590683c6 
  ql/src/test/results/clientpositive/dynpart_sort_opt_vectorization.q.out 
d03bfe422743d9a5a6b85f9a6198e1e27024f129 
  ql/src/test/results/clientpositive/dynpart_sort_optimization.q.out 
dec872ab0eef54bd92d5c2bc068e2805cc14e272 
  ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
832580325873dee741ba86239ee571873994a808 
  ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
a90e3f63b4646cf0ade9785a501ebd1a6b2a3406 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization.q.out 
723e8192f2735059005fc3c5c96732a2c4be49c1 

Diff: https://reviews.apache.org/r/47554/diff/


Testing
---


Thanks,

Jesús Camacho Rodríguez

[jira] [Created] (HIVE-13786) Fix the unit test failure org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle

2016-05-18 Thread Aihua Xu (JIRA)

Aihua Xu created HIVE-13786:
---

 Summary: Fix the unit test failure 
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
 Key: HIVE-13786
 URL: https://issues.apache.org/jira/browse/HIVE-13786
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Affects Versions: 2.1.0
Reporter: Aihua Xu
Assignee: Aihua Xu






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 46690: HIVE-13068

2016-05-18 Thread Ashutosh Chauhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46690/#review133793
---



Took another pass on golden file updates. Some of the changes may not be kosher 
but those you might already have covered in follow-up. Can you take a look to 
see if there is anything which is not covered in follow-up ?


ql/src/test/results/clientpositive/annotate_stats_filter.q.out (line 303)


Not propagating zip is null to select op?



ql/src/test/results/clientpositive/cbo_rp_udaf_percentile_approx_23.q.out (line 
527)


special case for NaN ?



ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out (line 1070)


Here we lost propgation. Already covered with one of the follow-ups?



ql/src/test/results/clientpositive/dynpart_sort_optimization.q.out (line 2297)


Here also we lost propagation. Covered with one of the follow-ups?



ql/src/test/results/clientpositive/filter_cond_pushdown.q.out (line 305)


Here also we lost propagation. Covered with one of the follow-ups?



ql/src/test/results/clientpositive/fold_case.q.out (line 414)


case folding missed.



ql/src/test/results/clientpositive/folder_predicate.q.out (line 40)


Number of expression evaluation increased to 4 from 3. Possible to bring it 
down?



ql/src/test/results/clientpositive/input26.q.out (line 47)


Here also we lost propagation. Covered with one of the follow-ups?



ql/src/test/results/clientpositive/input_part10.q.out (line 58)


Shuffling extra columns. Covered with existing columns?



ql/src/test/results/clientpositive/insert_into5.q.out (line 52)


Shuffling extra columns. Covered with follow-up jiras?



ql/src/test/results/clientpositive/join_filters_overlap.q.out (line 59)


Covered with one of the follow-ups?



ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out (line 
3010)


Always true?



ql/src/test/results/clientpositive/llap/tez_self_join.q.out (line 153)


Extra columns in shuffle?



ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
 (line 2765)


Always true?



ql/src/test/results/clientpositive/mergejoin.q.out (line 2702)


Extra partition retrieved for execution?



ql/src/test/results/clientpositive/pcr.q.out (line 2343)


Extra column shuffled.



ql/src/test/results/clientpositive/pointlookup3.q.out (line 477)


Extra columns.



ql/src/test/results/clientpositive/ppd_outer_join4.q.out (line 52)


Extra column for shuffle.



ql/src/test/results/clientpositive/ppd_outer_join5.q.out (line 53)


Propgating constant from FIL to SEL.



ql/src/test/results/clientpositive/rand_partitionpruner3.q.out (line 156)


4 expression evaluations instead of 3.



ql/src/test/results/clientpositive/semijoin4.q.out (line 76)


Extra columns referenced.



ql/src/test/results/clientpositive/union_view.q.out (line 716)


Extra (constant) column for shuffle. Propagation broken?



ql/src/test/results/clientpositive/vector_coalesce.q.out (line 209)


No propagation across RS?



ql/src/test/results/clientpositive/vector_decimal_2.q.out (line 919)


Is this change correct?



ql/src/test/results/clientpositive/vector_decimal_2.q.out (line 1023)


Is this change correct?



ql/src/test/results/clientpositive/vector_decimal_2.q.out (line 1075)


Is this change correct?



ql/src/test/results/clientpositive/vector_decimal_round_2.q.out (line 441)


Extra columns for shuffle.


- As

[jira] [Created] (HIVE-13785) Investigate recent test failures.

2016-05-18 Thread Aihua Xu (JIRA)

Aihua Xu created HIVE-13785:
---

 Summary: Investigate recent test failures.
 Key: HIVE-13785
 URL: https://issues.apache.org/jira/browse/HIVE-13785
 Project: Hive
  Issue Type: Test
Reporter: Aihua Xu


Seems at least the following tests are failing recently. 

 org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_22 
57 sec  3
 
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
1 min 3 sec 3
 org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries  
16 min  14
 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload 5.7 sec 
21
 org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec  
54 ms   21
 
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle 
   15 sec  21
 
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
   2.5 sec 21
 
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
5.6 sec 21




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Becoming a contributor

2016-05-18 Thread Vihang Karajgaonkar

Thanks Alan! My JIRA id is vihangk1

-Vihang

> On May 18, 2016, at 1:09 PM, Alan Gates  wrote:
> 
> Nope, you’re good.  You can ignore the stuff about review board as we 
> generally only use that now for large and complex patches.  If you have a 
> JIRA in mind you’d like to work on that no one else has you can assign it to 
> yourself* and get started.  If someone else is working on it you should 
> coordinate via comments on that JIRA to see if there’s a way you can help.
> 
> Welcome to the team.
> 
> Alan.
> 
> *You probably can’t assign JIRAs to yourself yet, but if you reply to this 
> email with your JIRA id I’ll make it so you can.
> 
> Alan.
> 
>> On May 18, 2016, at 11:42, Vihang Karajgaonkar  wrote:
>> 
>> Hello Everyone,
>> 
>> I would like to start working on issues reported on Hive in JIRA. I followed 
>> the steps mentioned in 
>> https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-BecomingaContributor
>>  
>> .
>>  Is there anything else that I need to do to get added as a contributor?
>> 
>> Thanks,
>> Vihang
>

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

2016-05-18 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46956/
---

(Updated May 18, 2016, 8:36 p.m.)


Review request for hive, Gunther Hagleitner, Jason Dere, and Siddharth Seth.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cbb3a72 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenLocalClient.java
 f10351b 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapSigner.java 
PRE-CREATION 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenIdentifier.java
 e28eddd 
  
llap-common/src/java/org/apache/hadoop/hive/llap/security/LlapTokenProvider.java
 PRE-CREATION 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
465b204 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 2524dc2 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java 
de817e3 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java
 b94fc2e 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTokenChecker.java
 03ee055 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
8abd198 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java
 eac0e8f 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
 74359fa 
  
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java
 PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSignerImpl.java 
PRE-CREATION 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 279baf1 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapTokenChecker.java
 762 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/comparator/TestFirstInFirstOutComparator.java
 a250882 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java c9b912b 

Diff: https://reviews.apache.org/r/46956/diff/


Testing
---


Thanks,

Sergey Shelukhin

Re: Becoming a contributor

2016-05-18 Thread Alan Gates

Nope, you’re good.  You can ignore the stuff about review board as we generally 
only use that now for large and complex patches.  If you have a JIRA in mind 
you’d like to work on that no one else has you can assign it to yourself* and 
get started.  If someone else is working on it you should coordinate via 
comments on that JIRA to see if there’s a way you can help.

Welcome to the team.

Alan.

*You probably can’t assign JIRAs to yourself yet, but if you reply to this 
email with your JIRA id I’ll make it so you can.

Alan.

> On May 18, 2016, at 11:42, Vihang Karajgaonkar  wrote:
> 
> Hello Everyone,
> 
> I would like to start working on issues reported on Hive in JIRA. I followed 
> the steps mentioned in 
> https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-BecomingaContributor
>  
> .
>  Is there anything else that I need to do to get added as a contributor?
> 
> Thanks,
> Vihang

Review Request 47546: HIVE-13448 LLAP: check ZK acls for ZKSM and fail if they are too permissive

2016-05-18 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47546/
---

Review request for hive, Prasanth_J and Siddharth Seth.


Repository: hive-git


Description
---


Diffs
-

  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 cffa493 
  llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java 
465b204 

Diff: https://reviews.apache.org/r/47546/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-13783) No secondary prompt

2016-05-18 Thread Vihang Karajgaonkar (JIRA)

Vihang Karajgaonkar created HIVE-13783:
--

 Summary: No secondary prompt
 Key: HIVE-13783
 URL: https://issues.apache.org/jira/browse/HIVE-13783
 Project: Hive
  Issue Type: Improvement
  Components: Beeline
Affects Versions: 2.0.0
Reporter: Vihang Karajgaonkar
Priority: Minor


{noformat}
# beeline -u jdbc:hive2://localhost:1
[...]
Beeline version 1.1.0-cdh5.4.5 by Apache Hive
0: jdbc:hive2://localhost:1> "
0: jdbc:hive2://localhost:1> select * from foo;
Error: Error while compiling statement: FAILED: ParseException line 2:17 
character '' not supported here (state=42000,code=4)
0: jdbc:hive2://localhost:1> 
{noformat}

After (accidentally) entering a lonely quote character on its own line and 
pressing Enter, I get back the normal prompt. This easily makes me believe I'm 
about to type a new command from scratch, e.g. a select query as in the 
example, which ends up not working due to parsing error.

Expected behavior: When a previous command is continued, or a quote is opened 
or anything like this, a differently looking secondary prompt should be 
displayed rather than the normal prompt; as this is done in e.g. hive, impala, 
mysql, bash..., e.g.:

{noformat}
# beeline -u jdbc:hive2://localhost:1
[...]
Beeline version 1.1.0-cdh5.4.5 by Apache Hive
0: jdbc:hive2://localhost:1> "
   > ...
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13784) Hive Metastore start failed on Oracle DB

2016-05-18 Thread Hari Sankar Sivarama Subramaniyan (JIRA)

Hari Sankar Sivarama Subramaniyan created HIVE-13784:


 Summary: Hive Metastore start failed on Oracle DB
 Key: HIVE-13784
 URL: https://issues.apache.org/jira/browse/HIVE-13784
 Project: Hive
  Issue Type: Bug
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan


During hive metastore start, hive is trying to execute this script 
{{hive-schema-2.1.0.oracle.sql}}:
{code}
Metastore connection URL:jdbc:oracle:thin:@//172.22.114.220:1521/XE
Metastore Connection Driver :oracle.jdbc.driver.OracleDriver
Metastore connection User:   hiveuser
Starting metastore schema initialization to 2.1.0
Initialization script hive-schema-2.1.0.oracle.sql
Connecting to jdbc:oracle:thin:@//172.22.114.220:1521/XE
Connected to: Oracle (version Oracle Database 11g Express Edition Release 
11.2.0.2.0 - 64bit Production)
Driver: Oracle JDBC driver (version 11.2.0.2.0)
Transaction isolation: TRANSACTION_READ_COMMITTED
0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> !autocommit on
Autocommit status: true
0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE TABLE SEQUENCE_TABLE ( 
SEQ 
UENCE_NAME VARCHAR2(255) NOT NULL, NEXT_VAL NUMBER NOT NULL )
No rows affected (0.208 seconds)
..
{code}

but inthe end of this script we are getting Error:
{code}
0: jdbc:oracle:thin:@//172.22.114.220:1521/XE> CREATE INDEX 
CONSTRAINTS_PARENT_T 
BL_ID_INDEX ON KEY_CONSTRAINTS(PARENT_TBL_ID)
Error: ORA-00972: identifier is too long (state=42000,code=972)

Closing: 0: jdbc:oracle:thin:@//172.22.114.220:1521/XE
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization 
FAILED! Metastore state would be inconsistent !!
org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization 
FAILED! Metastore state would be inconsistent !!
at 
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:291)
at 
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:264)
at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:505)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at 
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:390)
at 
org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:347)
at 
org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:287)
... 8 more
*** schemaTool failed ***
{code}
We have a problem with too long identifier: {{CONSTRAINTS_PARENT_TBL_ID_INDEX}} 
(31 symbol), it should be 30 symbols of max length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Becoming a contributor

2016-05-18 Thread Vihang Karajgaonkar

Hello Everyone,

I would like to start working on issues reported on Hive in JIRA. I followed 
the steps mentioned in 
https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-BecomingaContributor
 
.
 Is there anything else that I need to do to get added as a contributor?

Thanks,
Vihang

[jira] [Created] (HIVE-13782) Compile async query asynchronously

2016-05-18 Thread Jimmy Xiang (JIRA)

Jimmy Xiang created HIVE-13782:
--

 Summary: Compile async query asynchronously
 Key: HIVE-13782
 URL: https://issues.apache.org/jira/browse/HIVE-13782
 Project: Hive
  Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


Currently, when an async query is submitted to HS2, HS2 does the preparation 
synchronously. One of the preparation step is to compile the query, which may 
take some time. It will be helpful to provide an option to do the compilation 
asynchronously.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 46690: HIVE-13068

2016-05-18 Thread Ashutosh Chauhan



> On May 16, 2016, 4:30 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/constprog3.q.out, line 56
> > 
> >
> > Is this change correct?
> 
> Jesús Camacho Rodríguez wrote:
> This change is correct; observe that the other join input has a filter 
> with condition 'false'.
> 
> This is expected, as we were being too aggressive previously and had to 
> revert some changes in HiveRelMdPredicates that went in HIVE-13102 and that 
> could lead to incorrect inference of predicates for outer joins (basically, I 
> did not realize a common method was used for inner/outer joins).
> 
> However, observe that in this particular case, the original plan was 
> correct and better, as data is filtered out at the beginning of the plan in 
> both inputs. I can tackle this kind of inference for _inner_ joins in a 
> follow-up.

yeah.. its ok to do this in a follow-up. Please create a follow-up jira for it. 
It will be good to have this.


> On May 16, 2016, 4:30 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/cp_sel.q.out, line 98
> > 
> >
> > Constant propagation for partition column?
> 
> Jesús Camacho Rodríguez wrote:
> I think the problem here is not with the partition columns. Observe that 
> the original query is 
> _insert overwrite table testpartbucket partition(ds,hr) select 
> key,value,'hello' as ds, 'world' as hr from srcpart where hr=11;_
> 
> The problem is that CBO optimizes the _select_ query and then the 
> _insert_ part of the query is attached; constantpropagate in Hive does not 
> kick in anymore because CBO optimized the plan.

I see. Please create a follow-up jira for this. We may not tackle this right 
away but will be good to keep track of the issue.


> On May 16, 2016, 4:30 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/index_auto_unused.q.out, line 369
> > 
> >
> > no constant folding for 12 = 12?
> 
> Jesús Camacho Rodríguez wrote:
> Solved. The problem was that when we were applying PCR, we were not 
> updating the condition that we had pushed to the TableScan.

Cool.


> On May 16, 2016, 4:30 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/vector_decimal_2.q.out, line 928
> > 
> >
> > constant propagation?
> 
> Jesús Camacho Rodríguez wrote:
> Limitation in the original HiveSortLimitPullUpConstantsRule rule. The 
> problem is that Calcite rule does not pull-up constants when the Limit 
> operator is on top of the operator tree, as this was causing Hive limit 
> related optimizations to not kick in. I think we can solve this in a 
> follow-up, as it needs some investigation.

Ok. Lets do this in a follow-up.


> On May 16, 2016, 4:30 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/vector_null_projection.q.out, line 114
> > 
> >
> > null constant?
> 
> Jesús Camacho Rodríguez wrote:
> The Calcite rule that pull up constants through Aggregate leaves always 
> at least one column in the Aggregate and then it references it with a Project 
> on top. I could extend the rule to pull up the constant i.e. even if there is 
> a single column left, instead of referencing it on top, introduce a constant. 
> However, this means we need to bring the complete code of the rule to Hive... 
> do you think is it worth it? Or we extend the Calcite rule and the change 
> will come once we upgrade to the new Calcite version?

I think it will be good to have this even if we have to copy-paste Calcite 
rule, since shuffling on constants provide additional opportunities to optimize 
in ReduceSinkDedup. However, its ok to do it in a follow-up.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46690/#review133370
---


On May 18, 2016, 2:06 p.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46690/
> ---
> 
> (Updated May 18, 2016, 2:06 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-13068
> https://issues.apache.org/jira/browse/HIVE-13068
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13068
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_ppd_key_range.q.out 
> 27446b41db80ee98d56a4101a87f76be7f6dea2f 
>   ql/src/java/org/apache/hadoop/h

[jira] [Created] (HIVE-13781) Tez Job failed with FileNotFoundException when partition dir doesnt exists

2016-05-18 Thread Feng Yuan (JIRA)

Feng Yuan created HIVE-13781:


 Summary: Tez Job failed with FileNotFoundException when partition 
dir doesnt exists 
 Key: HIVE-13781
 URL: https://issues.apache.org/jira/browse/HIVE-13781
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Affects Versions: 0.14.0
 Environment: hive 0.14.0 ,tez-0.5.2,hadoop 2.6.0
Reporter: Feng Yuan


when i have a partitioned table a with partition "day",in metadata a have 
partition day: 20160501,20160502,but partition 20160501's dir didnt exits.
so when i use tez engine to run hive -e "select day,count(*) from a where xx=xx 
group by day"
hive throws FileNotFoundException.
but mr work.

repo eg:
CREATE EXTERNAL TABLE `a`(
  `a` string)
PARTITIONED BY ( 
  `l_date` string);

insert overwrite table a partition(l_date='2016-04-08') values (1),(2);
insert overwrite table a partition(l_date='2016-04-09') values (1),(2);
hadoop dfs -rm -r -f /warehouse/a/l_date=2016-04-09

select l_date,count(*) from a where a='1' group by l_date;

error:
ut: a initializer failed, vertex=vertex_1463493135662_10445_1_00 [Map 1], 
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: 
hdfs://bfdhadoopcool/warehouse/test.db/a/l_date=2015-04-09
at 
org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:285)
at 
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)
at 
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:300)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:402)
at 
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:129)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:245)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:239)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:239)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 46690: HIVE-13068

2016-05-18 Thread Jesús Camacho Rodríguez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46690/
---

(Updated May 18, 2016, 2:06 p.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-13068
https://issues.apache.org/jira/browse/HIVE-13068


Repository: hive-git


Description
---

HIVE-13068


Diffs (updated)
-

  hbase-handler/src/test/results/positive/hbase_ppd_key_range.q.out 
27446b41db80ee98d56a4101a87f76be7f6dea2f 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/NonBlockingOpDeDupProc.java 
37dbe32008685ba22e5dae1e4bfbfe090c5bfe9f 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
bf9a0a367b3b85f039076ac78290f8e35a8c3c62 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java
 010c89ed978296709b052cc7bc80256a27658e2b 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java 
0cfd5298899ea8dd16c073b26546c40de4451271 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOptUtil.java 
c6d1d46c62d8550750eea092245a55dd3b327f66 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRexUtil.java 
2f309f3de6acfac09b7b0d84cbb9d4275e317aeb 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregatePullUpConstantsRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectFilterPullUpConstantsRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveReduceExpressionsRule.java
 2fe9b75038de8261fa123aa6e1d318ea6b0d1cec 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortLimitPullUpConstantsRule.java
 3be9b0a0dafde81692db696f1a8f9099a132aec6 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveUnionPullUpConstantsRule.java
 2552f8747ba4b3d4f46d1d06a5fe381cbd039468 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdPredicates.java
 e8107471eaebaf95aeb32fa93b2917861ebb0795 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
 de7e2f8feae424a27075b17ad9fb7de2dd81e735 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ExprNodeConverter.java
 e51b6c49f447d04fdcac6d23deda5d980f43822d 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverterPostProc.java
 368264c1de1b406a76dd9e12848c0f8a94b0df54 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
 ee4f4ead6066a29e867cf51582c45d3dc69b1880 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
 0b76bffb42d88204f486278a12bbf24d1b7fc274 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
 2825f7787de4d42e9532bfb2642f4f95ba8f8b83 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
991117945e8bce1c4098f0641ff7674c8a314147 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrOpProcFactory.java 
65505b324e054617a2e87a233a54551b2d20c5d4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
de6a053a5b299ee39ec9af865d077a886497189f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
7162c089cd125c660abaad5838da28ab167c73b5 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
2eaed564304f0f8293ce35227fcfef15398305ef 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDescUtils.java 
c6f89074457e1ed4e61d52c01d9cc515fe1a6f09 
  ql/src/test/queries/clientpositive/join_view.q 
16b6816f0c8618691ba7a28f4ca467d7526d6e13 
  ql/src/test/results/clientpositive/annotate_stats_filter.q.out 
ba0419e461a5b5649bd7d4c67602b8cb747961ea 
  ql/src/test/results/clientpositive/archive_excludeHadoop20.q.out 
c2b98727d21f4990ae7496a0a8fa9ac16598f4c0 
  ql/src/test/results/clientpositive/archive_multi.q.out 
0ad29d122153bd4adf4d19064188b0c4f94e05ab 
  ql/src/test/results/clientpositive/authorization_explain.q.java1.7.out 
a9ed0495fcecadbddf1fcfb764e916fbb5406662 
  ql/src/test/results/clientpositive/auto_join33.q.out 
b0b3019d5c7a6ff6058b5bfd7c965257f8850367 
  ql/src/test/results/clientpositive/auto_join8.q.out 
324f95d550add0ead3215bbdd0933ddd6456f9c9 
  ql/src/test/results/clientpositive/auto_join_filters.q.out 
2fdf470036e0df898ad2986f3a26628e6e6bba44 
  ql/src/test/results/clientpositive/auto_join_nulls.q.out 
4af5535f4a0a9c07aca7342a0a31ddb9c9b2790f 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 
d8eacbeec54db58ddd2c6b194d6f012ad19a3bbc 
  ql/src/test/results/clientpositive/bucket_groupby.q.out 
ae736f98a19d4fa9a04fb9ba2f8fbe8c468d468a 
  ql/src/test/results/clientpositive/cast1.q.out 
48a0c14031ef38dd5b4df7efa718a4d6ce04bc94 
  ql/src/test/results/clientpositive/cbo_rp_join1.q.out 
f3982b8866d44dd0b4c01bf1f75e44c861f85756 
  ql/src/test/results/clientpositive/cbo_rp_udaf_percentile_approx_23.q.out 
8b8df12a85c8e5457269

Re: Review Request 46690: HIVE-13068

2016-05-18 Thread Jesús Camacho Rodríguez

> On May 16, 2016, 4:30 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/constprog3.q.out, line 56
> > 
> >
> > Is this change correct?

This change is correct; observe that the other join input has a filter with 
condition 'false'.

This is expected, as we were being too aggressive previously and had to revert 
some changes in HiveRelMdPredicates that went in HIVE-13102 and that could lead 
to incorrect inference of predicates for outer joins (basically, I did not 
realize a common method was used for inner/outer joins).

However, observe that in this particular case, the original plan was correct 
and better, as data is filtered out at the beginning of the plan in both 
inputs. I can tackle this kind of inference for _inner_ joins in a follow-up.

> On May 16, 2016, 4:30 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/cp_sel.q.out, line 98
> > 
> >
> > Constant propagation for partition column?

I think the problem here is not with the partition columns. Observe that the 
original query is 
_insert overwrite table testpartbucket partition(ds,hr) select 
key,value,'hello' as ds, 'world' as hr from srcpart where hr=11;_

The problem is that CBO optimizes the _select_ query and then the _insert_ part 
of the query is attached; constantpropagate in Hive does not kick in anymore 
because CBO optimized the plan.

> On May 16, 2016, 4:30 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/vector_decimal_2.q.out, line 928
> > 
> >
> > constant propagation?

Limitation in the original HiveSortLimitPullUpConstantsRule rule. The problem 
is that Calcite rule does not pull-up constants when the Limit operator is on 
top of the operator tree, as this was causing Hive limit related optimizations 
to not kick in. I think we can solve this in a follow-up, as it needs some 
investigation.

> On May 16, 2016, 4:30 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/vector_null_projection.q.out, line 114
> > 
> >
> > null constant?

The Calcite rule that pull up constants through Aggregate leaves always at 
least one column in the Aggregate and then it references it with a Project on 
top. I could extend the rule to pull up the constant i.e. even if there is a 
single column left, instead of referencing it on top, introduce a constant. 
However, this means we need to bring the complete code of the rule to Hive... 
do you think is it worth it? Or we extend the Calcite rule and the change will 
come once we upgrade to the new Calcite version?

> On May 16, 2016, 4:30 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/vector_interval_arithmetic.q.out, line 
> > 495
> > 
> >
> > constant propgation broken?

Same as comment above.

> On May 16, 2016, 4:30 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/index_auto_unused.q.out, line 369
> > 
> >
> > no constant folding for 12 = 12?

Solved. The problem was that when we were applying PCR, we were not updating 
the condition that we had pushed to the TableScan.

- Jesús

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46690/#review133370
---

On May 14, 2016, 11:47 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/46690/
> ---
> 
> (Updated May 14, 2016, 11:47 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-13068
> https://issues.apache.org/jira/browse/HIVE-13068
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13068
> 
> 
> Diffs
> -
> 
>   hbase-handler/src/test/results/positive/hbase_ppd_key_range.q.out 
> 27446b41db80ee98d56a4101a87f76be7f6dea2f 
>   hbase-handler/src/test/results/positive/hbase_pushdown.q.out 
> a42e36f0132457d22240d3ef4b8a52375a14fe34 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/NonBlockingOpDeDupProc.java 
> 37dbe32008685ba22e5dae1e4bfbfe090c5bfe9f 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 
> bf9a0a367b3b85f039076ac78290f8e35a8c3c62 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java
>  010c89ed978296709b052cc7bc80256a27658e2b 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOpti

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-05-18 Thread Takanobu Asanuma


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
---

(Updated 5月 18, 2016, 9:54 a.m.)


Review request for hive.


Changes
---

I updated the patch based on Sergey's advice. Also I merged the latest master 
branch.


Repository: hive-git


Description
---

This is a WIP patch for HIVE-11527

* I added a new configuration whose name is 
hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
true, clients use the bypass.

* I still have not considered security such as Kerberos and SSL at present.

* I have not implement Statement#setFetchSize for bypass yet.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cbb3a72 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 
84644d1 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
815ccfa 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
637e51a 
  jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca 
  jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 38ccc78 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3fecc5c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815 
  service-rpc/if/TCLIService.thrift 9879b1b 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 838bf17 
  service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 5229230 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
 31472c8 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
 7101fa5 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
 1aa3f94 
  
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
 14d50ed 
  service-rpc/src/gen/thrift/gen-php/Types.php 9ed7403 
  service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py 44e5462 
  service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb b39ec1e 
  service/src/java/org/apache/hive/service/cli/CLIService.java ed52b4a 
  service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java bfd7135 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 0932884 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
2f18231 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
67e0e52 
  service/src/java/org/apache/hive/service/cli/session/HiveSession.java 78ff388 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
a0015eb 
  service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
5464e58 

Diff: https://reviews.apache.org/r/40867/diff/


Testing
---

I have tested few simple queries and they worked well. But I think there are 
some problems for some queries. I'm going to test more queries and fix bugs. 
I'm also going to add unit tests.


Thanks,

Takanobu Asanuma

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

2016-05-18 Thread Takanobu Asanuma



> On 4月 22, 2016, 1:33 a.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 565
> > 
> >
> > nit: is it enough to just use Arrays.asList?

That's right. I will use just List here.


> On 4月 22, 2016, 1:33 a.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 588
> > 
> >
> > why does this swallow the error? It should fail on error

This is my mistake. I think it should not use a try-catch statement.


- Takanobu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review130014
---


On 4月 15, 2016, 11:38 a.m., Takanobu Asanuma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> ---
> 
> (Updated 4月 15, 2016, 11:38 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This is a WIP patch for HIVE-11527
> 
> * I added a new configuration whose name is 
> hive.server2.webhdfs.bypass.enabled. The default is false. When this value is 
> true, clients use the bypass.
> 
> * I still have not considered security such as Kerberos and SSL at present.
> 
> * I have not implement Statement#setFetchSize for bypass yet.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c7e5b33 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 
> 84644d1 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 857805a 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
> 637e51a 
>   jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209 
>   jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 0bbd0e3 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 92c2c76 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336 
>   service-rpc/if/TCLIService.thrift aa28b6e 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 7f1d9dd 
>   service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 3a27a60 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java
>  31472c8 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java
>  7101fa5 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java
>  1aa3f94 
>   
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java
>  14d50ed 
>   service-rpc/src/gen/thrift/gen-php/Types.php b7df50a 
>   service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py c691781 
>   service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 07ed97c 
>   service/src/java/org/apache/hive/service/cli/CLIService.java ab30ae2 
>   service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> b7d6549 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> 56a9c18 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 9ce6055 
>   service/src/java/org/apache/hive/service/cli/session/HiveSession.java 
> 4f4e92d 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 0f36cd6 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> be9833d 
> 
> Diff: https://reviews.apache.org/r/40867/diff/
> 
> 
> Testing
> ---
> 
> I have tested few simple queries and they worked well. But I think there are 
> some problems for some queries. I'm going to test more queries and fix bugs. 
> I'm also going to add unit tests.
> 
> 
> Thanks,
> 
> Takanobu Asanuma
> 
>

[jira] [Created] (HIVE-13788) hive msck listpartitions need to make use of directSQL instead of datanucleus

[jira] [Created] (HIVE-13787) LLAP: bug in recent security patches

Hive system catalog

Re: Becoming a contributor

Re: Review Request 47554: HIVE-13750

Re: Review Request 47419: enable merging of bit vectors for insert into

Re: Review Request 47419: enable merging of bit vectors for insert into

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

Review Request 47554: HIVE-13750

[jira] [Created] (HIVE-13786) Fix the unit test failure org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle

Re: Review Request 46690: HIVE-13068

[jira] [Created] (HIVE-13785) Investigate recent test failures.

Re: Becoming a contributor

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

Re: Becoming a contributor

Review Request 47546: HIVE-13448 LLAP: check ZK acls for ZKSM and fail if they are too permissive

[jira] [Created] (HIVE-13783) No secondary prompt

[jira] [Created] (HIVE-13784) Hive Metastore start failed on Oracle DB

Becoming a contributor

[jira] [Created] (HIVE-13782) Compile async query asynchronously

Re: Review Request 46690: HIVE-13068

[jira] [Created] (HIVE-13781) Tez Job failed with FileNotFoundException when partition dir doesnt exists

Re: Review Request 46690: HIVE-13068

Re: Review Request 46690: HIVE-13068

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift interface for query results

27 matches

Site Navigation

Mail list logo

Footer information