Re: Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59325/#review175192
---




llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
Lines 490 (patched)


Does the size() check need to happen within the lock - what if size() == 0, 
but a new fragment is added while we try to get the dagLock?


- Jason Dere


On May 17, 2017, 12:30 a.m., Siddharth Seth wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59325/
> ---
> 
> (Updated May 17, 2017, 12:30 a.m.)
> 
> 
> Review request for hive, Jason Dere and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-14052
> https://issues.apache.org/jira/browse/HIVE-14052
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Cleanup of structures required when LLAP access from external clients 
> completes
> 
> 
> Diffs
> -
> 
>   
> llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
>  ece31ed 
>   llap-common/src/protobuf/LlapDaemonProtocol.proto 3a3a2b8 
>   llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
> ce2f457 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  daeb555 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  27c426c 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> 868eec7 
> 
> 
> Diff: https://reviews.apache.org/r/59325/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Siddharth Seth
> 
>

Re: Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59325/#review175184
---




llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
Lines 192 (patched)


external tasks cannot use shuffle? just checking



llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
Lines 464 (patched)


will there be a callable created after every single fragment that finishes 
for the query with no other fragments running? Perhaps one callable should wait 
for the entire query and the fragments coming it should just push back the 
timestamp at which the query would time out and be cleaned?

also what if readLock blocks all cleanup attempts?


- Sergey Shelukhin


On May 17, 2017, 12:30 a.m., Siddharth Seth wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59325/
> ---
> 
> (Updated May 17, 2017, 12:30 a.m.)
> 
> 
> Review request for hive, Jason Dere and Sergey Shelukhin.
> 
> 
> Bugs: HIVE-14052
> https://issues.apache.org/jira/browse/HIVE-14052
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Cleanup of structures required when LLAP access from external clients 
> completes
> 
> 
> Diffs
> -
> 
>   
> llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
>  ece31ed 
>   llap-common/src/protobuf/LlapDaemonProtocol.proto 3a3a2b8 
>   llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
> ce2f457 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  daeb555 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  27c426c 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
> 868eec7 
> 
> 
> Diff: https://reviews.apache.org/r/59325/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Siddharth Seth
> 
>

[jira] [Created] (HIVE-16691) Add test for more datatypes for LlapInputFormat

[Jason Dere (JIRA)]

Jason Dere created HIVE-16691:
-

 Summary: Add test for more datatypes for LlapInputFormat
 Key: HIVE-16691
 URL: https://issues.apache.org/jira/browse/HIVE-16691
 Project: Hive
  Issue Type: Sub-task
  Components: llap
Reporter: Jason Dere
Assignee: Jason Dere


Update the test to include more data types.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size

[Zhiyuan Yang (JIRA)]

Zhiyuan Yang created HIVE-16690:
---

 Summary: Configure Tez cartesian product edge based on LLAP 
cluster size
 Key: HIVE-16690
 URL: https://issues.apache.org/jira/browse/HIVE-16690
 Project: Hive
  Issue Type: Bug
Reporter: Zhiyuan Yang
Assignee: Zhiyuan Yang


In HIVE-14731 we are using default value for target parallelism of fair 
cartesian product edge. Ideally this should be set according to cluster size. 
In case of LLAP it's pretty easy to get cluster size, i.e., number of executors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 56140: Can't order by an unselected column

[Ashutosh Chauhan]

> On May 3, 2017, 4:24 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/cp_sel.q.out
> > Line 46 (original), 50 (patched)
> > 
> >
> > Is this expected? 
> > Seems like this may generate wrong results since there might be 
> > multiple tasks for Reducers each of which emit 1 row. Limit in fetch 
> > operator is needed.
> 
> pengcheng xiong wrote:
> Yes, it is. In case of order by, only 1 reducer is used, so no need of 
> another shuffle.

Correct. Whats the reason for this? Is that because we got rid of Order by on 
Calcite tree itself?


> On May 3, 2017, 4:24 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/vector_coalesce.q.out
> > Line 447 (original)
> > 
> >
> > No RS for order by.
> 
> pengcheng xiong wrote:
> This is actually improvement. The query is {SELECT cfloat, cbigint, 
> coalesce(cfloat, cbigint, 0) as c
> FROM alltypesorc
> WHERE (cfloat IS NULL AND cbigint IS NULL)
> ORDER BY cfloat, cbigint, c
> LIMIT 10;}
> You can see that, cfloat, cbigint, c are all nulls
> 
> The op tree is like this
> 
> HiveSortLimit(offset=[0], fetch=[10])
>   HiveProject(cfloat=[$0], cbigint=[$1], c=[$2])
> HiveSortLimit(sort0=[$0], sort1=[$1], sort2=[$2], 
> dir0=[ASC-nulls-first], dir1=[ASC-nulls-first], dir2=[ASC-nulls-first])
>   HiveProject(cfloat=[$4], cbigint=[$3], c=[coalesce($4, $3, 0)], 
> ctinyint=[$0], csmallint=[$1], cint=[$2], cbigint1=[$3], cfloat1=[$4], 
> cdouble=[$5], cstring1=[$6], cstring2=[$7], ctimestamp1=[$8], 
> ctimestamp2=[$9], cboolean1=[$10], cboolean2=[$11], 
> block__offset__inside__file=[$12], input__file__name=[$13], row__id=[$14])
> HiveFilter(condition=[AND(IS NULL($4), IS NULL($3))])
>   HiveTableScan(table=[[default.alltypesorc]], 
> table:alias=[alltypesorc])
> 
> 
> After running HiveProjectFilterPullUpConstantsRule and 
> HiveReduceExpressionsRule, we get rid of the order by...

There is limit also in query. After this change, limit is executed in map side 
(potentially multiple tasks) but since FetchOperator doesnt have a limit, limit 
may not be honored.


> On May 3, 2017, 4:24 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/vector_date_1.q.out
> > Lines 598-607 (original)
> > 
> >
> > This plan looks incorrect. For an order by there should necessarily be 
> > a RS in plan, otherwise we can get sorting in map only plan.
> 
> pengcheng xiong wrote:
> dt1 is constant.

Is this change because we optimized away order by on calcite tree?


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56140/#review173690
---


On May 1, 2017, 5:30 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56140/
> ---
> 
> (Updated May 1, 2017, 5:30 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15160
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  1487ed4f8e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 262dafb487 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 654f3b1772 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 8f8eab0d9c 
>   ql/src/test/queries/clientpositive/order_by_expr_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/order_by_expr_2.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1abb25 
>   ql/src/test/results/clientpositive/cp_sel.q.out 1778ccd6a6 
>   ql/src/test/results/clientpositive/druid_basic2.q.out 6177d56987 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 2abb819558 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_grouping.q.out 
> 473d17a1bd 
>   ql/src/test/results/clientpositive/llap/bucket_groupby.q.out d724131fca 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 584c3b5520 
>   ql/src/test/results/clientpositive/llap/limit_pushdown.q.out dd54dd22a6 
>   ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out 24645b6426 
>   ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
> 83de1fbea1 
>   ql/src/test/results/clientpositive/llap/vector_coalesce.q.out 578f849bdb 
>   

[jira] [Created] (HIVE-16689) Correlated scalar subquery with comparison to constant in predicate fails

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-16689:
--

 Summary: Correlated scalar subquery with comparison to constant in 
predicate fails
 Key: HIVE-16689
 URL: https://issues.apache.org/jira/browse/HIVE-16689
 Project: Hive
  Issue Type: Bug
Reporter: Vineet Garg
Assignee: Vineet Garg


*Reproducer*
{code:sql}
CREATE TABLE `item`(
  `i_item_sk` int,
  `i_item_id` char(16),
  `i_rec_start_date` date,
  `i_rec_end_date` date,
  `i_item_desc` varchar(200),
  `i_current_price` decimal(7,2),
  `i_wholesale_cost` decimal(7,2),
  `i_brand_id` int,
  `i_brand` char(50),
  `i_class_id` int,
  `i_class` char(50),
  `i_category_id` int,
  `i_category` char(50),
  `i_manufact_id` int,
  `i_manufact` char(50),
  `i_size` char(20),
  `i_formulation` char(20),
  `i_color` char(20),
  `i_units` char(10),
  `i_container` char(10),
  `i_manager_id` int,
  `i_product_name` char(50));

select count(*)
 from item i1
 where
   (select count(*)
   from item
   where (i_manufact = i1.i_manufact)) > 0;
{code}

*Error stack*
{code}
org.apache.calcite.util.mapping.Mappings$NoElementException: source #0 has no 
target in mapping [size=0, sourceCount=1, targetCount=1, elements=[]]
at 
org.apache.calcite.util.mapping.Mappings$AbstractMapping.getTarget(Mappings.java:874)
 ~[calcite-core-1.12.0.jar:1.12.0]
at 
org.apache.calcite.sql2rel.RelFieldTrimmer$2.handle(RelFieldTrimmer.java:304) 
~[calcite-core-1.12.0.jar:1.12.0]
at 
org.apache.calcite.sql2rel.CorrelationReferenceFinder$MyRexVisitor.visitFieldAccess(CorrelationReferenceFinder.java:59)
 ~[calcite-core-1.12.0.jar:1.12.0]
at 
org.apache.calcite.sql2rel.CorrelationReferenceFinder$MyRexVisitor.visitFieldAccess(CorrelationReferenceFinder.java:50)
 ~[calcite-core-1.12.0.jar:1.12.0]
at org.apache.calcite.rex.RexFieldAccess.accept(RexFieldAccess.java:81) 
~[calcite-core-1.12.0.jar:1.12.0]
at org.apache.calcite.rex.RexShuttle.visitList(RexShuttle.java:148) 
~[calcite-core-1.12.0.jar:1.12.0]
at org.apache.calcite.rex.RexShuttle.visitCall(RexShuttle.java:97) 
~[calcite-core-1.12.0.jar:1.12.0]
at org.apache.calcite.rex.RexShuttle.visitCall(RexShuttle.java:36) 
~[calcite-core-1.12.0.jar:1.12.0]
at org.apache.calcite.rex.RexCall.accept(RexCall.java:104) 
~[calcite-core-1.12.0.jar:1.12.0]
at org.apache.calcite.rex.RexShuttle.apply(RexShuttle.java:279) 
~[calcite-core-1.12.0.jar:1.12.0]
at org.apache.calcite.rel.core.Filter.accept(Filter.java:103) 
~[calcite-core-1.12.0.jar:1.12.0]
at 
org.apache.calcite.sql2rel.CorrelationReferenceFinder.visit(CorrelationReferenceFinder.java:44)
 ~[calcite-core-1.12.0.jar:1.12.0]
at 
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter.accept(HiveFilter.java:116)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:55) 
~[calcite-core-1.12.0.jar:1.12.0]
at 
org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:69) 
~[calcite-core-1.12.0.jar:1.12.0]
at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:131) 
~[calcite-core-1.12.0.jar:1.12.0]
at 
org.apache.calcite.sql2rel.CorrelationReferenceFinder.visit(CorrelationReferenceFinder.java:43)
 ~[calcite-core-1.12.0.jar:1.12.0]
at 
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.accept(HiveProject.java:198)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:55) 
~[calcite-core-1.12.0.jar:1.12.0]
at 
org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:69) 
~[calcite-core-1.12.0.jar:1.12.0]
at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:131) 
~[calcite-core-1.12.0.jar:1.12.0]
at 
org.apache.calcite.sql2rel.CorrelationReferenceFinder.visit(CorrelationReferenceFinder.java:43)
 ~[calcite-core-1.12.0.jar:1.12.0]
at 
org.apache.calcite.rel.AbstractRelNode.accept(AbstractRelNode.java:279) 
~[calcite-core-1.12.0.jar:1.12.0]
at 
org.apache.calcite.rel.RelShuttleImpl.visitChild(RelShuttleImpl.java:55) 
~[calcite-core-1.12.0.jar:1.12.0]
at 
org.apache.calcite.rel.RelShuttleImpl.visitChildren(RelShuttleImpl.java:69) 
~[calcite-core-1.12.0.jar:1.12.0]
at org.apache.calcite.rel.RelShuttleImpl.visit(RelShuttleImpl.java:131) 
~[calcite-core-1.12.0.jar:1.12.0]
at 
org.apache.calcite.sql2rel.CorrelationReferenceFinder.visit(CorrelationReferenceFinder.java:43)
 ~[calcite-core-1.12.0.jar:1.12.0]
at 
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject.accept(HiveProject.java:198)
 ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 

Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

[Siddharth Seth]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59325/
---

Review request for hive, Jason Dere and Sergey Shelukhin.


Bugs: HIVE-14052
https://issues.apache.org/jira/browse/HIVE-14052


Repository: hive-git


Description
---

Cleanup of structures required when LLAP access from external clients completes


Diffs
-

  
llap-common/src/gen/protobuf/gen-java/org/apache/hadoop/hive/llap/daemon/rpc/LlapDaemonProtocolProtos.java
 ece31ed 
  llap-common/src/protobuf/LlapDaemonProtocol.proto 3a3a2b8 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
ce2f457 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
daeb555 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 27c426c 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 
868eec7 


Diff: https://reviews.apache.org/r/59325/diff/1/


Testing
---


Thanks,

Siddharth Seth

[jira] [Created] (HIVE-16688) Make sure Alter Table to set transaction=true acquires X lock

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-16688:
-

 Summary: Make sure Alter Table to set transaction=true acquires X 
lock
 Key: HIVE-16688
 URL: https://issues.apache.org/jira/browse/HIVE-16688
 Project: Hive
  Issue Type: Sub-task
  Components: Transactions
Affects Versions: 1.3.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


suppose we have non-acid table with some data
An insert op starts (long running)
An alter table runs to add (transactional=true)
An update is run which will read the list of "original" files and assign IDs on 
the fly which are written to a delta file.
The long running insert completes.
Another update is run which now sees a different set of "original" files and 
will (most likely) assign different IDs.

Need to make sure to mutex this



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16687) LLAP: Hash table loader memory estimation should consider current state of cluster

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-16687:


 Summary: LLAP: Hash table loader memory estimation should consider 
current state of cluster
 Key: HIVE-16687
 URL: https://issues.apache.org/jira/browse/HIVE-16687
 Project: Hive
  Issue Type: Bug
  Components: llap
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran


Hash table loader memory monitoring currently doesn't take into account the 
state of the cluster (other fragments loading hash tables, overall overall 
memory usage etc.). At current state, even when there is enough memory in the 
daemon, a query can get killed as it had exceed the no conditional task size 
for llap. 

This is follow up for HIVE-16582. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16686) repli invocations of distcp needs additional handling

[Sushanth Sowmyan (JIRA)]

Sushanth Sowmyan created HIVE-16686:
---

 Summary: repli invocations of distcp needs additional handling
 Key: HIVE-16686
 URL: https://issues.apache.org/jira/browse/HIVE-16686
 Project: Hive
  Issue Type: Sub-task
  Components: repl
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan


When REPL LOAD invokes distcp, there needs to be a way for the user invoking 
REPL LOAD to pass on arguments to distcp. In addition, there is sometimes a 
need for distcp to be invoked from within an impersonated context, such as 
running as user "hdfs", asking distcp to preserve ownerships of individual 
files.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16685) LLAP: consider adding a background thread to defragment cache

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-16685:
---

 Summary: LLAP: consider adding a background thread to defragment 
cache
 Key: HIVE-16685
 URL: https://issues.apache.org/jira/browse/HIVE-16685
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


See HIVE-16233; this will be split from that patch, since for now with cache 
always being full it may not be so useful with heavy workloads



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58601: HIVE-16233 llap: Query failed with AllocatorOutOfMemoryException

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58601/
---

(Updated May 16, 2017, 9:53 p.m.)


Review request for hive, Gopal V and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1c37b6e091 
  llap-server/runtest.sh PRE-CREATION 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
302918aadf 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapAllocatorBuffer.java 
PRE-CREATION 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapCacheableBuffer.java 
5c0b6f3d07 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapDataBuffer.java 
7d5c1014fb 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
23796f65e8 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheMemoryManager.java
 2132574fb5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java 
fd9d9420b9 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
 761fd00443 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
 3973c8a3aa 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/MemoryManager.java 
0f4d3c01d7 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/SerDeLowLevelCacheImpl.java
 cd5bc9b0b4 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleAllocator.java 
51eb34e949 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleBufferManager.java 
af7cf3d5f1 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
294fb2b41f 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 03bc3ced35 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 907200a6e3 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/VectorDeserializeOrcWriter.java
 c9df7d94a0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
 20af0d0b49 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
 2c7a234a08 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java
 1f3f7ea05e 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestBuddyAllocator.java 
a6080e63fa 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestBuddyAllocatorForceEvict.java
 PRE-CREATION 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
 e95f807679 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelLrfuCachePolicy.java
 0cce624682 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
 3059382942 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
5b2e9b5b87 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapDecider.java 
c81131e014 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java a9c1e61ba9 
  storage-api/src/java/org/apache/hadoop/hive/common/io/Allocator.java 
16b9713661 
  storage-api/src/java/org/apache/hadoop/hive/common/io/DataCache.java 
e53b737910 


Diff: https://reviews.apache.org/r/58601/diff/4/

Changes: https://reviews.apache.org/r/58601/diff/3-4/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-16684) Bootstrap REPL DUMP shouldn't fail when table is dropped after fetching the table names.

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-16684:
---

 Summary: Bootstrap REPL DUMP shouldn't fail when table is dropped 
after fetching the table names.
 Key: HIVE-16684
 URL: https://issues.apache.org/jira/browse/HIVE-16684
 Project: Hive
  Issue Type: Sub-task
  Components: Hive, repl
Affects Versions: 2.1.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan


Currently, bootstrap dump will fail if the table does't exist when try to dump. 
This shall occur when table is dropped after REPL DUMP fetched the table names.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 59205: HIVE-16579: CachedStore: improvements to partition col stats caching and cache column stats for unpartitioned table

[Vaibhav Gumashta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
---

(Updated May 16, 2017, 5:59 p.m.)


Review request for hive, Daniel Dai and Thejas Nair.


Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-16579


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1c37b6e 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 91a3a38 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
b96c27e 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
870896c 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f 
  metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690 
  metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java 
fcf6f27 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 
668499b 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
1cc838f 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 
7beee42 
  metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
0c7d8bb 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
 da6cd46 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java
 PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 f613c30 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 1720e37 
  
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 
0ab20d6 


Diff: https://reviews.apache.org/r/59205/diff/10/

Changes: https://reviews.apache.org/r/59205/diff/9-10/


Testing
---


Thanks,

Vaibhav Gumashta

[jira] [Created] (HIVE-16683) ORC WriterVersion gets ArrayIndexOutOfBoundsException on newer ORC files

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-16683:


 Summary: ORC WriterVersion gets ArrayIndexOutOfBoundsException on 
newer ORC files
 Key: HIVE-16683
 URL: https://issues.apache.org/jira/browse/HIVE-16683
 Project: Hive
  Issue Type: Bug
  Components: ORC
Affects Versions: 2.1.1, 2.2.0
Reporter: Owen O'Malley
Assignee: Owen O'Malley


This only impacts branch-2.1 and branch-2.2, because it has been fixed in the 
ORC project's code base via ORC-125.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 59205: HIVE-16579: CachedStore: improvements to partition col stats caching and cache column stats for unpartitioned table

[Thejas Nair]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/#review175116
---




metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
Line 266 (original)


don't we need this removePartitionColStatsFromCache call ?


- Thejas Nair


On May 16, 2017, 8:10 a.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59205/
> ---
> 
> (Updated May 16, 2017, 8:10 a.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-16579
> https://issues.apache.org/jira/browse/HIVE-16579
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-16579
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1c37b6e 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  91a3a38 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> b96c27e 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 870896c 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> b28983f 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java 
> fcf6f27 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 
> 668499b 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
> 1cc838f 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 
> 7beee42 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> 0c7d8bb 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
>  da6cd46 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java
>  PRE-CREATION 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  f613c30 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  1720e37 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  0ab20d6 
> 
> 
> Diff: https://reviews.apache.org/r/59205/diff/9/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[jira] [Created] (HIVE-16682) Check if the console message from the hive schema tool needs to print to logging file

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-16682:
---

 Summary: Check if the console message from the hive schema tool 
needs to print to logging file
 Key: HIVE-16682
 URL: https://issues.apache.org/jira/browse/HIVE-16682
 Project: Hive
  Issue Type: Sub-task
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Aihua Xu
Priority: Minor


>From HiveSchemaTool, most of the messages are printed to console and some of 
>them are printed to log. Evaluate the console messages if make sense to print 
>to log as well and what would be the best way to print them to avoid 
>duplication in case if LOG is configured to be console. 





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Review Request 59310: HIVE-15051 Test framework integration with findbugs, rat checks etc.

[Peter Vary]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59310/
---

Review request for hive, Akira Ajisaka, Marta Kuczora, Siddharth Seth, Thejas 
Nair, Vihang Karajgaonkar, and Barna Zsombor Klara.


Repository: hive-git


Description
---

Integrated Apache Yetus (https://yetus.apache.org/)
- Created hive personality to contain the hive specific configurations
- Created yetus-wrapper.sh to download yetus and patch the not yet released 
changes:
-- checkstyle_YETUS-484.sh
-- findbugs_YETUS-471.sh
-- maven_YETUS-506.sh
- Created test-patch.sh, and smart-apply-patch.sh to simplify testing/applying 
patch
- Added patchprocess directory to gitignore
- Two small modification in pom.xml
-- Adding findbugs configuration to build
-- Disabling failure on javadoc errors


Diffs
-

  .gitignore 47c59da 
  dev-support/checkstyle_YETUS-484.sh PRE-CREATION 
  dev-support/findbugs_YETUS-471.sh PRE-CREATION 
  dev-support/hive-personality.sh PRE-CREATION 
  dev-support/maven_YETUS-506.sh PRE-CREATION 
  dev-support/smart-apply-patch.sh PRE-CREATION 
  dev-support/test-patch.sh PRE-CREATION 
  dev-support/yetus-wrapper.sh PRE-CREATION 
  pom.xml 30fa50b 


Diff: https://reviews.apache.org/r/59310/diff/1/


Testing
---

To test a patch set the FINDBUGS_HOME, and go to the hive source root directory 
and run the test_patch.sh command like this:
./[petervary:~/dev/upstream/hive] $ export 
FINDBUGS_HOME=~/dev/upstream/findbugs-3.0.1/
./[petervary:~/dev/upstream/hive] $ ./dev-support/test-patch.sh 
~/Downloads/HIVE-16345.2.patch


Thanks,

Peter Vary

[jira] [Created] (HIVE-16681) beeline is jamed and

[xiaoqi (JIRA)]

xiaoqi created HIVE-16681:
-

 Summary: beeline is jamed and 
 Key: HIVE-16681
 URL: https://issues.apache.org/jira/browse/HIVE-16681
 Project: Hive
  Issue Type: Bug
  Components: Beeline, Metastore
Affects Versions: 0.13.0
Reporter: xiaoqi


The beeline is jamed  when i submit too many hive job .HiveServer2 and 
Metastore server still work ,Until I restart Metastore server  the beeline work 
.  The cluster isn't  block up,I still can submit MR job , It seems metastore 
influence use.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16680) explainanalyze_3.q and explainuser_3.q (MiniTezCliDriver) failing

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-16680:
--

 Summary: explainanalyze_3.q and explainuser_3.q (MiniTezCliDriver) 
failing
 Key: HIVE-16680
 URL: https://issues.apache.org/jira/browse/HIVE-16680
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Matt McCline


When I pushed HIVE-16602 to master, I regenerated these two q files and I 
pushed them with the patch:
ql/src/test/results/clientpositive/tez/explainanalyze_3.q.out
ql/src/test/results/clientpositive/tez/explainuser_3.q.out

However, I have just realized that those two tests had been failing for 
previous ptests runs and the change was not due to my patch.

I tracked the errors and they start failing with HIVE-16557 
(https://github.com/apache/hive/commit/6bfa2491bfa8b13626632c3b3ba7e87bcb0f04ae).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] hive pull request #184: HIVE-16678: Truncate on temporary table fails with t...

[sankarh]

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/184

HIVE-16678: Truncate on temporary table fails with table not found error.

Truncate table on temp table.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-16678

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/184.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #184


commit 56e4d6c78eb623b36d77e668b59bcc3917c6d822
Author: Sankar Hariappan 
Date:   2017-05-16T12:01:12Z

HIVE-16678: Truncate on temporary table fails with table not found error.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HIVE-16679) Missing ASF header on properties file in ptest2 project

[Barna Zsombor Klara (JIRA)]

Barna Zsombor Klara created HIVE-16679:
--

 Summary: Missing ASF header on properties file in ptest2 project
 Key: HIVE-16679
 URL: https://issues.apache.org/jira/browse/HIVE-16679
 Project: Hive
  Issue Type: Bug
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara
Priority: Trivial
 Fix For: 3.0.0


The ASF header is missing on 
{{testutils/ptest2//conf/deployed/master-mr2.properties}} causing the build of 
the ptest2 project to fail on a RAT check.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16678) Truncate on temporary table fails

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-16678:
---

 Summary: Truncate on temporary table fails
 Key: HIVE-16678
 URL: https://issues.apache.org/jira/browse/HIVE-16678
 Project: Hive
  Issue Type: Bug
  Components: Hive, HiveServer2
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan


Truncate table query on temporary table failing with error:
Error: Error while processing statement: FAILED: Execution Error, return code 1 
from org.apache.hadoop.hive.ql.exec.DDLTask. Exception while processing 
(state=08S01,code=1)
>From hsi logs:
{noformat}
NoSuchObjectException(message:default.s10k table not found)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(HiveMetaStore.java:2118)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.truncate_table(HiveMetaStore.java:2008)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy31.truncate_table(Unknown Source)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.truncateTable(HiveMetaStoreClient.java:1115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
at com.sun.proxy.$Proxy32.truncateTable(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2306)
at com.sun.proxy.$Proxy32.truncateTable(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.truncateTable(Hive.java:1217)
at 
org.apache.hadoop.hive.ql.exec.DDLTask.truncateTable(DDLTask.java:4462)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:556)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1905)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1607)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1354)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1123)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1116)
at 
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:242)
at 
org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
at 
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:334)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at 
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:348)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16677) CTAS with no data fails in Druid

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-16677:
--

 Summary: CTAS with no data fails in Druid
 Key: HIVE-16677
 URL: https://issues.apache.org/jira/browse/HIVE-16677
 Project: Hive
  Issue Type: Bug
  Components: Druid integration
Affects Versions: 3.0.0
Reporter: Jesus Camacho Rodriguez


If we create a table in Druid using a CTAS statement and the query executed to 
create the table produces no data, we fail with the following exception:

{noformat}
druid.DruidStorageHandler: Exception while commit
java.io.FileNotFoundException: File 
/tmp/workingDirectory/.staging-jcamachorodriguez_20170515053123_835c394b-2157-4f6b-bfed-a2753acd568e/segmentsDescriptorDir
 does not exist.
...
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16676) RENAME TABLE and RENAME PARTITION events shall be modified as DROP+CREATE events.

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-16676:
---

 Summary: RENAME TABLE and RENAME PARTITION events shall be 
modified as DROP+CREATE events.
 Key: HIVE-16676
 URL: https://issues.apache.org/jira/browse/HIVE-16676
 Project: Hive
  Issue Type: Sub-task
  Components: repl
Affects Versions: 2.1.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan


Currently, RENAME TABLE and RENAME PARTITION events are treated as ALTER 
events. 
For bootstrap dump, if the table is renamed after fetching the table names, 
then new table will be missing in the dump and so the target database doesn't 
have both old and new table. During incremental replication, later RENAME 
events will be noop as the old table doesn't exist in target.
In order to make RENAME replication simple, it is suggested to treat RENAME as 
DROP+CREATE event.
EVENT_RENAME_TABLE = EVENT_DROP_TABLE + EVENT_CREATE_TABLE.
EVENT_RENAME_PARTITION = EVENT_DROP_PARTITION + EVENT_ADD_PARTITION.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16675) Fix ConcurrentModificationException in SparkClientImpl#startDriver

[liyunzhang_intel (JIRA)]

liyunzhang_intel created HIVE-16675:
---

 Summary: Fix ConcurrentModificationException in 
SparkClientImpl#startDriver
 Key: HIVE-16675
 URL: https://issues.apache.org/jira/browse/HIVE-16675
 Project: Hive
  Issue Type: Bug
Reporter: liyunzhang_intel
Assignee: liyunzhang_intel


the exception is
{noformat}
  2017-05-16T00:29:37,480  WARN [Driver] client.SparkClientImpl: 
Exception while waiting for child process.
  3926 java.util.ConcurrentModificationException
  3927 at 
java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) ~[?:1.8.0_91]
  3928 at java.util.ArrayList$Itr.next(ArrayList.java:851) 
~[?:1.8.0_91]
  3929 at 
org.apache.hive.spark.client.SparkClientImpl$3.run(SparkClientImpl.java:495) 
[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
  3930 at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
{noformat}

It seems that {{SparkClientImpl.java#childErrorLog}} is read while it is 
written. It is better to change {{SparkClientImpl.java#childErrorLog}} from 
ArrayList to CopyOnWriteArrayList to avoid the exception.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 59205: HIVE-16579: CachedStore: improvements to partition col stats caching and cache column stats for unpartitioned table

[Vaibhav Gumashta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59205/
---

(Updated May 16, 2017, 8:10 a.m.)


Review request for hive, Daniel Dai and Thejas Nair.


Bugs: HIVE-16579
https://issues.apache.org/jira/browse/HIVE-16579


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-16579


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1c37b6e 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 91a3a38 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d296851 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
b96c27e 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
870896c 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b28983f 
  metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java c1af690 
  metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java 
fcf6f27 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 
668499b 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
1cc838f 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 
7beee42 
  metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
0c7d8bb 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
 da6cd46 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java
 PRE-CREATION 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 f613c30 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 1720e37 
  
metastore/src/test/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java 
0ab20d6 


Diff: https://reviews.apache.org/r/59205/diff/9/

Changes: https://reviews.apache.org/r/59205/diff/8-9/


Testing
---


Thanks,

Vaibhav Gumashta

[jira] [Created] (HIVE-16674) Hive metastore JVM dumps core

[Vlad Gudikov (JIRA)]

Vlad Gudikov created HIVE-16674:
---

 Summary: Hive metastore JVM dumps core
 Key: HIVE-16674
 URL: https://issues.apache.org/jira/browse/HIVE-16674
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.2.1
 Environment: Hive-1.2.1
Kerberos enabled cluster
Reporter: Vlad Gudikov


While trying to run a Hive query on 24 partitions executed on an external table 
with large amount of partitions (4K+). I get an error

 - org.apache.thrift.transport.TSaslTransport$SaslParticipant.wrap(byte[], int, 
int) @bci=27, line=568 (Compiled frame)
 - org.apache.thrift.transport.TSaslTransport.flush() @bci=52, line=492 
(Compiled frame)
 - org.apache.thrift.transport.TSaslServerTransport.flush() @bci=1, line=41 
(Compiled frame)
 - org.apache.thrift.ProcessFunction.process(int, 
org.apache.thrift.protocol.TProtocol, org.apache.thrift.protocol.TProtocol, 
java.lang.Object) @bci=236, line=55 (Compiled frame)
 - 
org.apache.thrift.TBaseProcessor.process(org.apache.thrift.protocol.TProtocol, 
org.apache.thrift.protocol.TProtocol) @bci=126, line=39 (Compiled frame)
 - 
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run()
 @bci=15, line=690 (Compiled frame)
 - 
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run()
 @bci=1, line=685 (Compiled frame)
 - 
java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction,
 java.security.AccessControlContext) @bci=0 (Compiled frame)
 - javax.security.auth.Subject.doAs(javax.security.auth.Subject, 
java.security.PrivilegedExceptionAction) @bci=42, line=422 (Compiled frame)
 - 
org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction)
 @bci=14, line=1595 (Compiled frame)
 - 
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(org.apache.thrift.protocol.TProtocol,
 org.apache.thrift.protocol.TProtocol) @bci=273, line=685 (Compiled frame)
 - org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() @bci=151, 
line=285 (Interpreted frame)
 - 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
 @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16673) Unit test for HIVE-16413

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-16673:
---

 Summary: Unit test for HIVE-16413
 Key: HIVE-16673
 URL: https://issues.apache.org/jira/browse/HIVE-16673
 Project: Hive
  Issue Type: Test
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16672) Parquet vectorization doesn't work for tables with partition info

[Colin Ma (JIRA)]

Colin Ma created HIVE-16672:
---

 Summary: Parquet vectorization doesn't work for tables with 
partition info
 Key: HIVE-16672
 URL: https://issues.apache.org/jira/browse/HIVE-16672
 Project: Hive
  Issue Type: Sub-task
Reporter: Colin Ma
Assignee: Colin Ma
Priority: Critical
 Fix For: 3.0.0


VectorizedParquetRecordReader doesn't check and update partition cols, this 
should be fixed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)