hive git commit: HIVE-9599 : remove derby, datanucleus and other not related to jdbc client classes from hive-jdbc-standalone.jar (Ashutosh Chauhan via Thejas Nair)

[hashutosh]

Repository: hive
Updated Branches:
  refs/heads/master b7281ce6a -> 1b6600de0


HIVE-9599 : remove derby, datanucleus and other not related to jdbc client 
classes from hive-jdbc-standalone.jar (Ashutosh Chauhan via Thejas Nair)

Signed-off-by: Ashutosh Chauhan 


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/1b6600de
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/1b6600de
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/1b6600de

Branch: refs/heads/master
Commit: 1b6600de0f908170061628a6b5ed4f072012cc96
Parents: b7281ce
Author: Ashutosh Chauhan 
Authored: Wed Nov 18 15:00:30 2015 -0800
Committer: Ashutosh Chauhan 
Committed: Tue Nov 24 15:06:23 2015 -0800

--
 jdbc/pom.xml | 52 
 1 file changed, 52 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/1b6600de/jdbc/pom.xml
--
diff --git a/jdbc/pom.xml b/jdbc/pom.xml
index ea961a4..f8d7dfd 100644
--- a/jdbc/pom.xml
+++ b/jdbc/pom.xml
@@ -189,6 +189,58 @@
   org.antlr:*
   org.slf4j:slf4j-log4j12
   log4j:*
+  antlr:*
+  aopalliance:*
+  asm:*
+  com.google.code.gson:*
+  com.google.inject:*
+  com.google.inject.extensions:*
+  com.jamesmurty.utils:*
+  com.jcraft:*
+  com.jolbox:*
+  commons-beanutils:*
+  commons-cli:*
+  commons-dbcp:*
+  commons-digester:*
+  commons-el:*
+  commons-httpclient:*
+  commons-io:*
+  commons-net:*
+  commons-pool:*
+  com.google.code.findbugs:*
+  com.google.protobuf:*
+  com.sun.xml.bind:*
+  com.thoughtworks.paranamer:*
+  com.twitter:*
+  io.netty:*
+  javax.activation:*
+  javax.inject:*
+  javax.jdo:*
+  javax.mail:*
+  javax.servlet:*
+  javax.servlet.jsp:*
+  javax.transaction:*
+  javax.xml.bind:*
+  javax.xml.stream:*
+  jline:*
+  joda-time:*
+  net.java.dev.jets3t:*
+  org.apache.commons:commons-math3
+  org.apache.curator:*
+  org.apache.derby:*
+  org.apache.directory.api:*
+  org.apache.directory.server:*
+  org.apache.geronimo.specs:*
+  org.apache.zookeeper:*
+  org.codehaus.jackson:*
+  org.codehaus.jettison:*
+  org.datanucleus:*
+  org.fusesource.leveldbjni:*
+  org.htrace:*
+  org.mortbay.jetty:*
+  org.xerial.snappy:*
+  tomcat:*
+  xmlenc:*
 
   
 

hive git commit: HIVE-9599 : remove derby, datanucleus and other not related to jdbc client classes from hive-jdbc-standalone.jar (Ashutosh Chauhan via Thejas Nair)

[omalley]

Repository: hive
Updated Branches:
  refs/heads/master-fixed adbc0ab6a -> 1918735ed


HIVE-9599 : remove derby, datanucleus and other not related to jdbc client 
classes from hive-jdbc-standalone.jar (Ashutosh Chauhan via Thejas Nair)

Signed-off-by: Ashutosh Chauhan 


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/1918735e
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/1918735e
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/1918735e

Branch: refs/heads/master-fixed
Commit: 1918735ed206aeafd8115b3d6ee7d7ecb1cd5f1b
Parents: adbc0ab
Author: Ashutosh Chauhan 
Authored: Wed Nov 18 15:00:30 2015 -0800
Committer: Owen O'Malley 
Committed: Tue Nov 24 15:16:43 2015 -0800

--
 jdbc/pom.xml | 52 
 1 file changed, 52 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/1918735e/jdbc/pom.xml
--
diff --git a/jdbc/pom.xml b/jdbc/pom.xml
index ea961a4..f8d7dfd 100644
--- a/jdbc/pom.xml
+++ b/jdbc/pom.xml
@@ -189,6 +189,58 @@
   org.antlr:*
   org.slf4j:slf4j-log4j12
   log4j:*
+  antlr:*
+  aopalliance:*
+  asm:*
+  com.google.code.gson:*
+  com.google.inject:*
+  com.google.inject.extensions:*
+  com.jamesmurty.utils:*
+  com.jcraft:*
+  com.jolbox:*
+  commons-beanutils:*
+  commons-cli:*
+  commons-dbcp:*
+  commons-digester:*
+  commons-el:*
+  commons-httpclient:*
+  commons-io:*
+  commons-net:*
+  commons-pool:*
+  com.google.code.findbugs:*
+  com.google.protobuf:*
+  com.sun.xml.bind:*
+  com.thoughtworks.paranamer:*
+  com.twitter:*
+  io.netty:*
+  javax.activation:*
+  javax.inject:*
+  javax.jdo:*
+  javax.mail:*
+  javax.servlet:*
+  javax.servlet.jsp:*
+  javax.transaction:*
+  javax.xml.bind:*
+  javax.xml.stream:*
+  jline:*
+  joda-time:*
+  net.java.dev.jets3t:*
+  org.apache.commons:commons-math3
+  org.apache.curator:*
+  org.apache.derby:*
+  org.apache.directory.api:*
+  org.apache.directory.server:*
+  org.apache.geronimo.specs:*
+  org.apache.zookeeper:*
+  org.codehaus.jackson:*
+  org.codehaus.jettison:*
+  org.datanucleus:*
+  org.fusesource.leveldbjni:*
+  org.htrace:*
+  org.mortbay.jetty:*
+  org.xerial.snappy:*
+  tomcat:*
+  xmlenc:*
 
   
 

[3/7] hive git commit: HIVE-12456: QueryId can't be stored in the configuration of the SessionState since multiple queries can run in a single session (Aihua Xu, reviewed by Mohit)

[omalley]

HIVE-12456: QueryId can't be stored in the configuration of the SessionState 
since multiple queries can run in a single session (Aihua Xu, reviewed by Mohit)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/f15d4e10
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/f15d4e10
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/f15d4e10

Branch: refs/heads/master-fixed
Commit: f15d4e108103fd2b1c42345634e167e41ded42f9
Parents: 5f726d5
Author: Aihua Xu 
Authored: Mon Nov 23 12:20:39 2015 -0500
Committer: Owen O'Malley 
Committed: Tue Nov 24 12:10:09 2015 -0800

--
 .../cli/operation/ExecuteStatementOperation.java | 15 +--
 .../hive/service/cli/operation/Operation.java| 19 +++
 .../hive/service/cli/operation/SQLOperation.java |  4 ++--
 .../service/cli/session/HiveSessionImpl.java |  1 -
 4 files changed, 18 insertions(+), 21 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/f15d4e10/service/src/java/org/apache/hive/service/cli/operation/ExecuteStatementOperation.java
--
diff --git 
a/service/src/java/org/apache/hive/service/cli/operation/ExecuteStatementOperation.java
 
b/service/src/java/org/apache/hive/service/cli/operation/ExecuteStatementOperation.java
index 3f2de10..b3d9b52 100644
--- 
a/service/src/java/org/apache/hive/service/cli/operation/ExecuteStatementOperation.java
+++ 
b/service/src/java/org/apache/hive/service/cli/operation/ExecuteStatementOperation.java
@@ -18,7 +18,6 @@
 package org.apache.hive.service.cli.operation;
 
 import java.sql.SQLException;
-import java.util.HashMap;
 import java.util.Map;
 
 import org.apache.hadoop.hive.ql.processors.CommandProcessor;
@@ -29,13 +28,11 @@ import org.apache.hive.service.cli.session.HiveSession;
 
 public abstract class ExecuteStatementOperation extends Operation {
   protected String statement = null;
-  protected Map confOverlay = new HashMap();
 
   public ExecuteStatementOperation(HiveSession parentSession, String statement,
   Map confOverlay, boolean runInBackground) {
-super(parentSession, OperationType.EXECUTE_STATEMENT, runInBackground);
+super(parentSession, confOverlay, OperationType.EXECUTE_STATEMENT, 
runInBackground);
 this.statement = statement;
-setConfOverlay(confOverlay);
   }
 
   public String getStatement() {
@@ -57,14 +54,4 @@ public abstract class ExecuteStatementOperation extends 
Operation {
 }
 return new HiveCommandOperation(parentSession, statement, processor, 
confOverlay);
   }
-
-  protected Map getConfOverlay() {
-return confOverlay;
-  }
-
-  protected void setConfOverlay(Map confOverlay) {
-if (confOverlay != null) {
-  this.confOverlay = confOverlay;
-}
-  }
 }

http://git-wip-us.apache.org/repos/asf/hive/blob/f15d4e10/service/src/java/org/apache/hive/service/cli/operation/Operation.java
--
diff --git 
a/service/src/java/org/apache/hive/service/cli/operation/Operation.java 
b/service/src/java/org/apache/hive/service/cli/operation/Operation.java
index d13415e..25cefc2 100644
--- a/service/src/java/org/apache/hive/service/cli/operation/Operation.java
+++ b/service/src/java/org/apache/hive/service/cli/operation/Operation.java
@@ -21,11 +21,14 @@ import java.io.File;
 import java.io.FileNotFoundException;
 import java.io.IOException;
 import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.Map;
 import java.util.Set;
 import java.util.concurrent.Future;
 import java.util.concurrent.TimeUnit;
 
 import com.google.common.collect.Sets;
+
 import org.apache.hadoop.hive.common.metrics.common.Metrics;
 import org.apache.hadoop.hive.common.metrics.common.MetricsConstant;
 import org.apache.hadoop.hive.common.metrics.common.MetricsFactory;
@@ -50,8 +53,8 @@ import org.apache.logging.log4j.ThreadContext;
 
 public abstract class Operation {
   // Constants of the key strings for the log4j ThreadContext.
-  private static final String QUERYID = "QueryId";
-  private static final String SESSIONID = "SessionId";
+  public static final String SESSIONID_LOG_KEY = "sessionId";
+  public static final String QUERYID_LOG_KEY = "queryId";
 
   protected final HiveSession parentSession;
   private OperationState state = OperationState.INITIALIZED;
@@ -67,6 +70,7 @@ public abstract class Operation {
   protected volatile Future backgroundHandle;
   protected OperationLog operationLog;
   protected boolean isOperationLogEnabled;
+  protected Map confOverlay = new HashMap();
 
   private long operationTimeout;
   private long lastAccessTime;

[1/7] hive git commit: HIVE-12411: Remove counter based stats collection mechanism (Pengcheng Xiong, reviewed by Ashutosh Chauhan)

[omalley]

Repository: hive
Updated Branches:
  refs/heads/master-fixed 5f726d58e -> adbc0ab6a


http://git-wip-us.apache.org/repos/asf/hive/blob/884ff9ca/ql/src/test/results/clientpositive/stats_counter_partitioned.q.out
--
diff --git a/ql/src/test/results/clientpositive/stats_counter_partitioned.q.out 
b/ql/src/test/results/clientpositive/stats_counter_partitioned.q.out
deleted file mode 100644
index 626dcff..000
--- a/ql/src/test/results/clientpositive/stats_counter_partitioned.q.out
+++ /dev/null
@@ -1,465 +0,0 @@
-PREHOOK: query: -- partitioned table analyze 
-
-create table dummy (key string, value string) partitioned by (ds string, hr 
string)
-PREHOOK: type: CREATETABLE
-PREHOOK: Output: database:default
-PREHOOK: Output: default@dummy
-POSTHOOK: query: -- partitioned table analyze 
-
-create table dummy (key string, value string) partitioned by (ds string, hr 
string)
-POSTHOOK: type: CREATETABLE
-POSTHOOK: Output: database:default
-POSTHOOK: Output: default@dummy
-PREHOOK: query: load data local inpath '../../data/files/kv1.txt' into table 
dummy partition (ds='2008',hr='12')
-PREHOOK: type: LOAD
- A masked pattern was here 
-PREHOOK: Output: default@dummy
-POSTHOOK: query: load data local inpath '../../data/files/kv1.txt' into table 
dummy partition (ds='2008',hr='12')
-POSTHOOK: type: LOAD
- A masked pattern was here 
-POSTHOOK: Output: default@dummy
-POSTHOOK: Output: default@dummy@ds=2008/hr=12
-PREHOOK: query: load data local inpath '../../data/files/kv1.txt' into table 
dummy partition (ds='2008',hr='11')
-PREHOOK: type: LOAD
- A masked pattern was here 
-PREHOOK: Output: default@dummy
-POSTHOOK: query: load data local inpath '../../data/files/kv1.txt' into table 
dummy partition (ds='2008',hr='11')
-POSTHOOK: type: LOAD
- A masked pattern was here 
-POSTHOOK: Output: default@dummy
-POSTHOOK: Output: default@dummy@ds=2008/hr=11
-PREHOOK: query: analyze table dummy partition (ds,hr) compute statistics
-PREHOOK: type: QUERY
-PREHOOK: Input: default@dummy
-PREHOOK: Input: default@dummy@ds=2008/hr=11
-PREHOOK: Input: default@dummy@ds=2008/hr=12
-PREHOOK: Output: default@dummy
-PREHOOK: Output: default@dummy@ds=2008/hr=11
-PREHOOK: Output: default@dummy@ds=2008/hr=12
-POSTHOOK: query: analyze table dummy partition (ds,hr) compute statistics
-POSTHOOK: type: QUERY
-POSTHOOK: Input: default@dummy
-POSTHOOK: Input: default@dummy@ds=2008/hr=11
-POSTHOOK: Input: default@dummy@ds=2008/hr=12
-POSTHOOK: Output: default@dummy
-POSTHOOK: Output: default@dummy@ds=2008/hr=11
-POSTHOOK: Output: default@dummy@ds=2008/hr=12
-PREHOOK: query: describe formatted dummy partition (ds='2008', hr='11')
-PREHOOK: type: DESCTABLE
-PREHOOK: Input: default@dummy
-POSTHOOK: query: describe formatted dummy partition (ds='2008', hr='11')
-POSTHOOK: type: DESCTABLE
-POSTHOOK: Input: default@dummy
-# col_name data_type   comment 
-
-keystring  
-value  string  
-
-# Partition Information 
-# col_name data_type   comment 
-
-ds string  
-hr string  
-
-# Detailed Partition Information
-Partition Value:   [2008, 11]   
-Database:  default  
-Table: dummy
- A masked pattern was here 
-Partition Parameters:   
-   COLUMN_STATS_ACCURATE   true
-   numFiles1   
-   numRows 500 
-   rawDataSize 5312
-   totalSize   5812
- A masked pattern was here 
-
-# Storage Information   
-SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe  
 
-InputFormat:   org.apache.hadoop.mapred.TextInputFormat 
-OutputFormat:  
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat   
-Compressed:No   
-Num Buckets:   -1   
-Bucket Columns:[]   
-Sort Columns:  []   
-Storage Desc Params:
-   serialization.format1   
-PREHOOK: query: describe formatted dummy partition (ds='2008', hr='12')
-PREHOOK: type: DESCTABLE
-PREHOOK: Input: default@dummy
-POSTHOOK: query: describe formatted dummy partition (ds='2008', hr='12')
-POSTHOOK: type: DESCTABLE
-POSTHOOK: Input: default@dummy
-# col_name data_type   comment 
-
-keystring 

[2/7] hive git commit: HIVE-12411: Remove counter based stats collection mechanism (Pengcheng Xiong, reviewed by Ashutosh Chauhan)

[omalley]

HIVE-12411: Remove counter based stats collection mechanism (Pengcheng Xiong, 
reviewed by Ashutosh Chauhan)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/884ff9ca
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/884ff9ca
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/884ff9ca

Branch: refs/heads/master-fixed
Commit: 884ff9caf4dff634a3501ac6f3daa689d1ce751a
Parents: 6e429d8
Author: Pengcheng Xiong 
Authored: Tue Nov 24 17:09:40 2015 +0800
Committer: Owen O'Malley 
Committed: Tue Nov 24 12:10:09 2015 -0800

--
 .../hadoop/hive/common/StatsSetupConst.java |  13 -
 .../org/apache/hadoop/hive/conf/HiveConf.java   |   8 +-
 data/conf/llap/hive-site.xml|   2 +-
 data/conf/spark/standalone/hive-site.xml|   2 +-
 data/conf/spark/yarn-client/hive-site.xml   |   2 +-
 data/conf/tez/hive-site.xml |   4 +-
 .../hive/ql/stats/CounterStatsAggregator.java   |  82 
 .../ql/stats/CounterStatsAggregatorSpark.java   |  58 ---
 .../ql/stats/CounterStatsAggregatorTez.java |  79 
 .../hive/ql/stats/CounterStatsPublisher.java|  66 ---
 .../hadoop/hive/ql/stats/StatsFactory.java  |  11 -
 .../test/queries/clientpositive/index_bitmap3.q |   1 -
 .../queries/clientpositive/index_bitmap_auto.q  |   1 -
 .../test/queries/clientpositive/stats_counter.q |  16 -
 .../clientpositive/stats_counter_partitioned.q  |  45 --
 .../clientpositive/llap/stats_counter.q.out | 102 
 .../llap/stats_counter_partitioned.q.out| 465 ---
 .../clientpositive/spark/stats_counter.q.out| 102 
 .../spark/stats_counter_partitioned.q.out   | 465 ---
 .../results/clientpositive/stats_counter.q.out  | 102 
 .../stats_counter_partitioned.q.out | 465 ---
 .../clientpositive/tez/metadataonly1.q.out  |  72 +--
 .../clientpositive/tez/optimize_nullscan.q.out  |  90 ++--
 .../clientpositive/tez/stats_counter.q.out  | 102 
 .../tez/stats_counter_partitioned.q.out | 465 ---
 25 files changed, 88 insertions(+), 2732 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/884ff9ca/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java
--
diff --git a/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 
b/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java
index 0a44bde..2ff76ee 100644
--- a/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java
+++ b/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java
@@ -31,19 +31,6 @@ import java.util.Map;
 public class StatsSetupConst {
 
   public enum StatDB {
-counter {
-  @Override
-  public String getPublisher(Configuration conf) {
-return "org.apache.hadoop.hive.ql.stats.CounterStatsPublisher"; }
-  @Override
-  public String getAggregator(Configuration conf) {
-if (HiveConf.getVar(conf, 
HiveConf.ConfVars.HIVE_EXECUTION_ENGINE).equals("tez")) {
-  return "org.apache.hadoop.hive.ql.stats.CounterStatsAggregatorTez";
-} else if (HiveConf.getVar(conf, 
HiveConf.ConfVars.HIVE_EXECUTION_ENGINE).equals("spark")) {
-  return "org.apache.hadoop.hive.ql.stats.CounterStatsAggregatorSpark";
-}
-return "org.apache.hadoop.hive.ql.stats.CounterStatsAggregator"; }
-},
 fs {
   @Override
   public String getPublisher(Configuration conf) {

http://git-wip-us.apache.org/repos/asf/hive/blob/884ff9ca/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
--
diff --git a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
index f48403b..fffedd9 100644
--- a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
+++ b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
@@ -1354,10 +1354,10 @@ public class HiveConf extends Configuration {
 // Statistics
 HIVESTATSAUTOGATHER("hive.stats.autogather", true,
 "A flag to gather statistics automatically during the INSERT OVERWRITE 
command."),
-HIVESTATSDBCLASS("hive.stats.dbclass", "fs", new PatternSet("counter", 
"custom", "fs"),
+HIVESTATSDBCLASS("hive.stats.dbclass", "fs", new PatternSet("custom", 
"fs"),
 "The storage that stores temporary Hive statistics. In filesystem 
based statistics collection ('fs'), \n" +
 "each task writes statistics it has collected in a file on the 
filesystem, which will be aggregated \n" +
-"after the job has finished. Supported values are fs (filesystem), 
counter, and custom as defined in 

[6/7] hive git commit: HIVE-12509: Regenerate q files after HIVE-12017 went in (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)

[omalley]

HIVE-12509: Regenerate q files after HIVE-12017 went in (Jesus Camacho 
Rodriguez, reviewed by Ashutosh Chauhan)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/e34588e1
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/e34588e1
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/e34588e1

Branch: refs/heads/master-fixed
Commit: e34588e1b2c7fca678ec47e4659aca2bbc0a2ce4
Parents: 884ff9c
Author: Jesus Camacho Rodriguez 
Authored: Tue Nov 24 17:08:20 2015 +0100
Committer: Owen O'Malley 
Committed: Tue Nov 24 12:10:10 2015 -0800

--
 .../results/clientpositive/subquery_notin.q.out | 328 ++-
 .../subquery_notin_having.q.java1.7.out |  94 +++---
 .../clientpositive/tez/explainuser_1.q.out  | 171 +-
 .../tez/tez_dynpart_hashjoin_3.q.out| 150 -
 .../clientpositive/tez/tez_smb_empty.q.out  |  38 ++-
 5 files changed, 398 insertions(+), 383 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/e34588e1/ql/src/test/results/clientpositive/subquery_notin.q.out
--
diff --git a/ql/src/test/results/clientpositive/subquery_notin.q.out 
b/ql/src/test/results/clientpositive/subquery_notin.q.out
index 552999d..56553fd 100644
--- a/ql/src/test/results/clientpositive/subquery_notin.q.out
+++ b/ql/src/test/results/clientpositive/subquery_notin.q.out
@@ -1,4 +1,4 @@
-Warning: Shuffle Join JOIN[23][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 
'Stage-2:MAPRED' is a cross product
+Warning: Shuffle Join JOIN[20][tables = [$hdt$_0, $hdt$_1]] in Stage 
'Stage-1:MAPRED' is a cross product
 PREHOOK: query: -- non agg, non corr
 explain
 select * 
@@ -18,70 +18,64 @@ where src.key not in
   )
 POSTHOOK: type: QUERY
 STAGE DEPENDENCIES:
-  Stage-1 is a root stage
-  Stage-2 depends on stages: Stage-1, Stage-4
-  Stage-4 is a root stage
+  Stage-3 is a root stage
+  Stage-1 depends on stages: Stage-3
+  Stage-2 depends on stages: Stage-1
   Stage-0 depends on stages: Stage-2
 
 STAGE PLANS:
-  Stage: Stage-1
+  Stage: Stage-3
 Map Reduce
   Map Operator Tree:
   TableScan
 alias: src
 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE 
Column stats: NONE
-Select Operator
-  expressions: key (type: string), value (type: string)
-  outputColumnNames: _col0, _col1
-  Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE 
Column stats: NONE
-  Reduce Output Operator
-key expressions: _col0 (type: string)
-sort order: +
-Map-reduce partition columns: _col0 (type: string)
-Statistics: Num rows: 500 Data size: 5312 Basic stats: 
COMPLETE Column stats: NONE
-value expressions: _col1 (type: string)
-  TableScan
-alias: src
-Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE 
Column stats: NONE
 Filter Operator
-  predicate: (key > '2') (type: boolean)
-  Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE 
Column stats: NONE
+  predicate: ((key > '2') and key is null) (type: boolean)
+  Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE 
Column stats: NONE
   Select Operator
-expressions: key (type: string)
-outputColumnNames: _col0
-Statistics: Num rows: 166 Data size: 1763 Basic stats: 
COMPLETE Column stats: NONE
-Reduce Output Operator
-  key expressions: _col0 (type: string)
-  sort order: +
-  Map-reduce partition columns: _col0 (type: string)
-  Statistics: Num rows: 166 Data size: 1763 Basic stats: 
COMPLETE Column stats: NONE
+Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE 
Column stats: NONE
+Group By Operator
+  aggregations: count()
+  mode: hash
+  outputColumnNames: _col0
+  Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE 
Column stats: NONE
+  Reduce Output Operator
+sort order: 
+Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE 
Column stats: NONE
+value expressions: _col0 (type: bigint)
   Reduce Operator Tree:
-Join Operator
-  condition map:
-   Left Outer Join0 to 1
-  keys:
-0 _col0 (type: string)
-1 _col0 (type: string)
-  outputColumnNames: _col0, _col1, _col2
-  Statistics: Num rows: 550 

[7/7] hive git commit: HIVE-12175: Upgrade Kryo version to 3.0.x (Prasanth Jayachandran reviewed by Ashutosh Chauhan)

[omalley]

HIVE-12175: Upgrade Kryo version to 3.0.x (Prasanth Jayachandran reviewed by 
Ashutosh Chauhan)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/adbc0ab6
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/adbc0ab6
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/adbc0ab6

Branch: refs/heads/master-fixed
Commit: adbc0ab6aeff848dbcee83d565febd40797300c2
Parents: e34588e
Author: Prasanth Jayachandran 
Authored: Tue Nov 24 12:43:46 2015 -0600
Committer: Owen O'Malley 
Committed: Tue Nov 24 12:10:10 2015 -0800

--
 itests/qtest-accumulo/pom.xml   |   2 +-
 pom.xml |   6 +-
 ql/pom.xml  |  36 +++--
 .../apache/hadoop/hive/ql/exec/Utilities.java   | 145 +--
 .../org/apache/hadoop/hive/ql/plan/MapWork.java |  15 --
 .../apache/hadoop/hive/ql/plan/ReduceWork.java  |   5 -
 spark-client/pom.xml|  28 ++--
 .../hive/spark/client/rpc/KryoMessageCodec.java |  11 +-
 8 files changed, 185 insertions(+), 63 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/adbc0ab6/itests/qtest-accumulo/pom.xml
--
diff --git a/itests/qtest-accumulo/pom.xml b/itests/qtest-accumulo/pom.xml
index 7403a15..f7325dc 100644
--- a/itests/qtest-accumulo/pom.xml
+++ b/itests/qtest-accumulo/pom.xml
@@ -123,7 +123,7 @@
 
 
-  com.esotericsoftware.kryo
+  com.esotericsoftware
   kryo
   ${kryo.version}
   test

http://git-wip-us.apache.org/repos/asf/hive/blob/adbc0ab6/pom.xml
--
diff --git a/pom.xml b/pom.xml
index c6df4a5..c38c10f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -144,7 +144,7 @@
 3.5.2
 20090211
 4.11
-2.22
+3.0.3
 0.9.3
 0.9.3
 2.4
@@ -228,8 +228,8 @@
 
   
   
-com.esotericsoftware.kryo
-kryo
+com.esotericsoftware
+kryo-shaded
 ${kryo.version}
   
   

http://git-wip-us.apache.org/repos/asf/hive/blob/adbc0ab6/ql/pom.xml
--
diff --git a/ql/pom.xml b/ql/pom.xml
index 9420a62..d893099 100644
--- a/ql/pom.xml
+++ b/ql/pom.xml
@@ -72,8 +72,8 @@
 
 
 
-  com.esotericsoftware.kryo
-  kryo
+  com.esotericsoftware
+  kryo-shaded
   ${kryo.version}
 
 
@@ -594,16 +594,20 @@
   spark-core_${scala.binary.version}
   ${spark.version}
   true
-
- 
-org.slf4j
-slf4j-log4j12
-  
-  
-commmons-logging
-commons-logging
-  
-
+  
+   
+ com.esotericsoftware.kryo
+ kryo
+   
+   
+ org.slf4j
+ slf4j-log4j12
+   
+   
+ commmons-logging
+ commons-logging
+   
+ 

 
   com.sun.jersey
@@ -746,7 +750,9 @@
   org.apache.hive:hive-serde
   org.apache.hive:hive-llap-client
   org.apache.hive:hive-metastore
-  com.esotericsoftware.kryo:kryo
+  com.esotericsoftware:kryo-shaded
+ com.esotericsoftware:minlog
+ org.objenesis:objenesis
   org.apache.parquet:parquet-hadoop-bundle
   org.apache.thrift:libthrift
   org.apache.thrift:libfb303
@@ -779,6 +785,10 @@
   com.esotericsoftware
   
org.apache.hive.com.esotericsoftware
 
+
+  org.objenesis
+  org.apache.hive.org.objenesis
+
   
 
   

http://git-wip-us.apache.org/repos/asf/hive/blob/adbc0ab6/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
--
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
index 9dbb45a..8b8cf6d 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
@@ -37,6 +37,8 @@ import java.io.InputStream;
 import java.io.OutputStream;
 import java.io.Serializable;
 import java.io.UnsupportedEncodingException;
+import java.lang.reflect.Array;
+import java.lang.reflect.Field;
 import java.net.URI;
 import java.net.URL;
 import java.net.URLClassLoader;
@@ -87,8 +89,6 @@ import org.apache.commons.lang.StringUtils;
 import org.apache.commons.lang.WordUtils;
 import org.apache.commons.lang3.StringEscapeUtils;
 import 

[4/7] hive git commit: HIVE-12406: HIVE-9500 introduced incompatible change to LazySimpleSerDe public interface (Aihua Xu, reviewed by Szehon Ho)

[omalley]

HIVE-12406: HIVE-9500 introduced incompatible change to LazySimpleSerDe public 
interface (Aihua Xu, reviewed by Szehon Ho)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/6e429d83
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/6e429d83
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/6e429d83

Branch: refs/heads/master-fixed
Commit: 6e429d8381333edccba23d74fecd7342a69d09c1
Parents: cf6fbbd
Author: Aihua Xu 
Authored: Mon Nov 23 14:41:58 2015 -0500
Committer: Owen O'Malley 
Committed: Tue Nov 24 12:10:09 2015 -0800

--
 .../hive/serde2/lazy/LazySerDeParameters.java   |  6 ++-
 .../hive/serde2/lazy/LazySimpleSerDe.java   | 38 +
 .../hive/serde2/lazy/TestLazySimpleSerDe.java   | 56 +---
 3 files changed, 82 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/6e429d83/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySerDeParameters.java
--
diff --git 
a/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySerDeParameters.java 
b/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySerDeParameters.java
index 54f6b2b..11af860 100644
--- a/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySerDeParameters.java
+++ b/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySerDeParameters.java
@@ -20,14 +20,14 @@ package org.apache.hadoop.hive.serde2.lazy;
 
 import java.util.ArrayList;
 import java.util.Arrays;
-import java.util.HashMap;
 import java.util.List;
-import java.util.Map;
 import java.util.Properties;
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.classification.InterfaceAudience.Public;
+import org.apache.hadoop.hive.common.classification.InterfaceStability.Stable;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.serde.serdeConstants;
 import org.apache.hadoop.hive.serde2.SerDeException;
@@ -42,6 +42,8 @@ import org.apache.hive.common.util.HiveStringUtils;
  * SerDeParameters.
  *
  */
+@Public
+@Stable
 public class LazySerDeParameters implements LazyObjectInspectorParameters {
   public static final Logger LOG = 
LoggerFactory.getLogger(LazySerDeParameters.class.getName());
   public static final byte[] DefaultSeparators = {(byte) 1, (byte) 2, (byte) 
3};

http://git-wip-us.apache.org/repos/asf/hive/blob/6e429d83/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java
--
diff --git 
a/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 
b/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java
index 0a2f44c..ac2d39f 100644
--- a/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java
+++ b/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java
@@ -19,16 +19,14 @@
 package org.apache.hadoop.hive.serde2.lazy;
 
 import java.io.IOException;
-import java.util.ArrayList;
 import java.util.Arrays;
-import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
 import java.util.Properties;
 
-import org.apache.commons.lang.ArrayUtils;
 import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.common.classification.InterfaceAudience.Public;
+import org.apache.hadoop.hive.common.classification.InterfaceStability.Stable;
 import org.apache.hadoop.hive.serde.serdeConstants;
 import org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe;
 import org.apache.hadoop.hive.serde2.ByteStream;
@@ -36,7 +34,6 @@ import org.apache.hadoop.hive.serde2.SerDeException;
 import org.apache.hadoop.hive.serde2.SerDeSpec;
 import org.apache.hadoop.hive.serde2.SerDeStats;
 import org.apache.hadoop.hive.serde2.SerDeUtils;
-import 
org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyObjectInspectorParameters;
 import 
org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyObjectInspectorParametersImpl;
 import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
 import org.apache.hadoop.hive.serde2.objectinspector.MapObjectInspector;
@@ -48,14 +45,10 @@ import 
org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
 import org.apache.hadoop.hive.serde2.objectinspector.UnionObjectInspector;
 import 
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
 import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo;
-import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
-import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
 import 

[5/7] hive git commit: HIVE-12489 : Analyze for partition fails if partition value has special characters (Thomas Friedrich via Ashutosh Chauhan)

[omalley]

HIVE-12489 : Analyze for partition fails if partition value has special 
characters (Thomas Friedrich via Ashutosh Chauhan)

Signed-off-by: Ashutosh Chauhan 


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/cf6fbbd2
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/cf6fbbd2
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/cf6fbbd2

Branch: refs/heads/master-fixed
Commit: cf6fbbd2da8f3eebf1054c3da1bb76b6cb540bd1
Parents: f15d4e1
Author: Thomas Friedrich 
Authored: Fri Nov 20 13:55:00 2015 -0800
Committer: Owen O'Malley 
Committed: Tue Nov 24 12:10:09 2015 -0800

--
 .../ql/parse/ColumnStatsSemanticAnalyzer.java   |  6 +--
 .../queries/clientpositive/analyze_tbl_part.q   | 12 +
 .../clientpositive/analyze_tbl_part.q.out   | 52 
 3 files changed, 67 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/cf6fbbd2/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java
--
diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
b/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java
index 543bc0f..832a5bc 100644
--- 
a/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java
+++ 
b/ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java
@@ -199,7 +199,7 @@ public class ColumnStatsSemanticAnalyzer extends 
SemanticAnalyzer {
 if (partColType.equals(serdeConstants.STRING_TYPE_NAME) ||
 partColType.contains(serdeConstants.VARCHAR_TYPE_NAME) ||
 partColType.contains(serdeConstants.CHAR_TYPE_NAME)) {
-  returnVal = "'" + partVal + "'";
+  returnVal = "'" + escapeSQLString(partVal) + "'";
 } else if (partColType.equals(serdeConstants.TINYINT_TYPE_NAME)) {
   returnVal = partVal+"Y";
 } else if (partColType.equals(serdeConstants.SMALLINT_TYPE_NAME)) {
@@ -212,10 +212,10 @@ public class ColumnStatsSemanticAnalyzer extends 
SemanticAnalyzer {
   returnVal = partVal + "BD";
 } else if (partColType.equals(serdeConstants.DATE_TYPE_NAME) ||
 partColType.equals(serdeConstants.TIMESTAMP_TYPE_NAME)) {
-  returnVal = partColType + " '" + partVal + "'";
+  returnVal = partColType + " '" + escapeSQLString(partVal) + "'";
 } else {
   //for other usually not used types, just quote the value
-  returnVal = "'" + partVal + "'";
+  returnVal = "'" + escapeSQLString(partVal) + "'";
 }
 
 return returnVal;

http://git-wip-us.apache.org/repos/asf/hive/blob/cf6fbbd2/ql/src/test/queries/clientpositive/analyze_tbl_part.q
--
diff --git a/ql/src/test/queries/clientpositive/analyze_tbl_part.q 
b/ql/src/test/queries/clientpositive/analyze_tbl_part.q
index c9e45b6..ecf1389 100644
--- a/ql/src/test/queries/clientpositive/analyze_tbl_part.q
+++ b/ql/src/test/queries/clientpositive/analyze_tbl_part.q
@@ -15,3 +15,15 @@ ANALYZE TABLE src_stat_part partition (partitionId) COMPUTE 
STATISTICS for colum
 describe formatted src_stat_part.key PARTITION(partitionId=1);
 
 describe formatted src_stat_part.value PARTITION(partitionId=2);
+
+create table src_stat_string_part(key string, value string) partitioned by 
(partitionName string);
+
+insert overwrite table src_stat_string_part partition (partitionName="p'1")
+select * from src1;
+
+insert overwrite table src_stat_string_part partition (partitionName="p\"1")
+select * from src1;
+
+ANALYZE TABLE src_stat_string_part partition (partitionName="p'1") COMPUTE 
STATISTICS for columns key, value;
+
+ANALYZE TABLE src_stat_string_part partition (partitionName="p\"1") COMPUTE 
STATISTICS for columns key, value;
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/hive/blob/cf6fbbd2/ql/src/test/results/clientpositive/analyze_tbl_part.q.out
--
diff --git a/ql/src/test/results/clientpositive/analyze_tbl_part.q.out 
b/ql/src/test/results/clientpositive/analyze_tbl_part.q.out
index 40b926c..464bdf7 100644
--- a/ql/src/test/results/clientpositive/analyze_tbl_part.q.out
+++ b/ql/src/test/results/clientpositive/analyze_tbl_part.q.out
@@ -81,3 +81,55 @@ POSTHOOK: Input: default@src_stat_part
 # col_name data_type   min max 
num_nulls   distinct_count  avg_col_len 
max_col_len num_trues   num_falses  
comment 

 
 value  string 

[2/4] hive git commit: HIVE-12331 : Remove hive.enforce.bucketing & hive.enforce.sorting configs (Ashutosh Chauhan via Jason Dere)

[hashutosh]

http://git-wip-us.apache.org/repos/asf/hive/blob/5562fae7/ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table.q.out
--
diff --git 
a/ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table.q.out 
b/ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table.q.out
deleted file mode 100644
index b1dfd7c..000
--- 
a/ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table.q.out
+++ /dev/null
@@ -1,553 +0,0 @@
-PREHOOK: query: -- INCLUDE_HADOOP_MAJOR_VERSIONS(0.20,0.20S)
-create table tst1(key string, value string) partitioned by (ds string) 
clustered by (key) into 10 buckets
-PREHOOK: type: CREATETABLE
-PREHOOK: Output: database:default
-POSTHOOK: query: -- INCLUDE_HADOOP_MAJOR_VERSIONS(0.20,0.20S)
-create table tst1(key string, value string) partitioned by (ds string) 
clustered by (key) into 10 buckets
-POSTHOOK: type: CREATETABLE
-POSTHOOK: Output: database:default
-POSTHOOK: Output: default@tst1
-PREHOOK: query: alter table tst1 clustered by (key) into 8 buckets
-PREHOOK: type: ALTERTABLE_CLUSTER_SORT
-PREHOOK: Input: default@tst1
-PREHOOK: Output: default@tst1
-POSTHOOK: query: alter table tst1 clustered by (key) into 8 buckets
-POSTHOOK: type: ALTERTABLE_CLUSTER_SORT
-POSTHOOK: Input: default@tst1
-POSTHOOK: Output: default@tst1
-PREHOOK: query: describe formatted tst1
-PREHOOK: type: DESCTABLE
-PREHOOK: Input: default@tst1
-POSTHOOK: query: describe formatted tst1
-POSTHOOK: type: DESCTABLE
-POSTHOOK: Input: default@tst1
-# col_name data_type   comment 
-
-keystring  
-value  string  
-
-# Partition Information 
-# col_name data_type   comment 
-
-ds string  
-
-# Detailed Table Information
-Database:  default  
- A masked pattern was here 
-Retention: 0
- A masked pattern was here 
-Table Type:MANAGED_TABLE
-Table Parameters:   
- A masked pattern was here 
-
-# Storage Information   
-SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe  
 
-InputFormat:   org.apache.hadoop.mapred.TextInputFormat 
-OutputFormat:  
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat   
-Compressed:No   
-Num Buckets:   8
-Bucket Columns:[key]
-Sort Columns:  []   
-Storage Desc Params:
-   serialization.format1   
-PREHOOK: query: insert overwrite table tst1 partition (ds='1') select key, 
value from src
-PREHOOK: type: QUERY
-PREHOOK: Input: default@src
-PREHOOK: Output: default@tst1@ds=1
-POSTHOOK: query: insert overwrite table tst1 partition (ds='1') select key, 
value from src
-POSTHOOK: type: QUERY
-POSTHOOK: Input: default@src
-POSTHOOK: Output: default@tst1@ds=1
-POSTHOOK: Lineage: tst1 PARTITION(ds=1).key SIMPLE 
[(src)src.FieldSchema(name:key, type:string, comment:default), ]
-POSTHOOK: Lineage: tst1 PARTITION(ds=1).value SIMPLE 
[(src)src.FieldSchema(name:value, type:string, comment:default), ]
-PREHOOK: query: describe formatted tst1 partition (ds = '1')
-PREHOOK: type: DESCTABLE
-PREHOOK: Input: default@tst1
-POSTHOOK: query: describe formatted tst1 partition (ds = '1')
-POSTHOOK: type: DESCTABLE
-POSTHOOK: Input: default@tst1
-# col_name data_type   comment 
-
-keystring  
-value  string  
-
-# Partition Information 
-# col_name data_type   comment 
-
-ds string  
-
-# Detailed Partition Information
-Partition Value:   [1]  
-Database:  default  
-Table: tst1 
- A masked pattern was here 
-Partition Parameters:   
-   COLUMN_STATS_ACCURATE   true
-   numFiles1   
-   numRows 500 
-   rawDataSize 5312
-   totalSize   5812
- A masked pattern was here 
-
-# Storage Information   
-SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe  
 
-InputFormat:   

[4/4] hive git commit: HIVE-12331 : Remove hive.enforce.bucketing & hive.enforce.sorting configs (Ashutosh Chauhan via Jason Dere)

[hashutosh]

HIVE-12331 : Remove hive.enforce.bucketing & hive.enforce.sorting configs 
(Ashutosh Chauhan via Jason Dere)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/5562fae7
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/5562fae7
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/5562fae7

Branch: refs/heads/master
Commit: 5562fae73e417c81a193c1e6deb6388d3fef746b
Parents: 1b6600d
Author: Ashutosh Chauhan 
Authored: Tue Nov 24 17:08:35 2015 -0800
Committer: Ashutosh Chauhan 
Committed: Tue Nov 24 17:08:35 2015 -0800

--
 .../org/apache/hadoop/hive/conf/HiveConf.java   |   8 +-
 .../org/apache/hadoop/hive/ql/ErrorMsg.java |   1 +
 .../apache/hadoop/hive/ql/exec/Utilities.java   |   3 +-
 .../optimizer/SortedDynPartitionOptimizer.java  |  86 +-
 .../hadoop/hive/ql/parse/SemanticAnalyzer.java  |  54 +-
 .../apache/hadoop/hive/ql/TestTxnCommands.java  |  14 +-
 .../apache/hadoop/hive/ql/TestTxnCommands2.java |   6 +-
 .../hive/ql/lockmgr/TestDbTxnManager2.java  |   5 +-
 .../queries/clientnegative/acid_overwrite.q |   2 +-
 .../queries/clientnegative/archive_corrupt.q|   2 +-
 .../authorization_delete_nodeletepriv.q |   2 +-
 .../authorization_update_noupdatepriv.q |   2 +-
 .../clientnegative/delete_non_acid_table.q  |   2 +-
 .../clientnegative/delete_not_bucketed.q|   2 +-
 .../test/queries/clientnegative/delete_sorted.q |   2 +-
 .../test/queries/clientnegative/insert_sorted.q |   2 +-
 .../clientnegative/insert_values_sorted.q   |   2 +-
 .../queries/clientnegative/merge_negative_3.q   |   4 +-
 .../queries/clientnegative/smb_bucketmapjoin.q  |   4 +-
 .../queries/clientnegative/smb_mapjoin_14.q |   4 +-
 .../sortmerge_mapjoin_mismatch_1.q  |   4 +-
 .../queries/clientnegative/update_bucket_col.q  |   2 +-
 .../clientnegative/update_no_such_table.q   |   2 +-
 .../clientnegative/update_non_acid_table.q  |   2 +-
 .../clientnegative/update_not_bucketed.q|   2 +-
 .../clientnegative/update_partition_col.q   |   2 +-
 .../test/queries/clientnegative/update_sorted.q |   2 +-
 ql/src/test/queries/clientpositive/acid_join.q  |   2 +-
 .../queries/clientpositive/acid_vectorization.q |   2 +-
 .../acid_vectorization_partition.q  |   2 +-
 .../clientpositive/acid_vectorization_project.q |   2 +-
 .../alter_numbuckets_partitioned_table.q|  59 --
 .../alter_numbuckets_partitioned_table2.q   |  85 --
 .../alter_numbuckets_partitioned_table2_h23.q   |   5 +-
 .../alter_numbuckets_partitioned_table_h23.q|   2 +-
 .../clientpositive/archive_excludeHadoop20.q|   2 +-
 .../test/queries/clientpositive/archive_multi.q |   2 +-
 .../clientpositive/authorization_delete.q   |   2 +-
 .../authorization_delete_own_table.q|   2 +-
 .../clientpositive/authorization_update.q   |   2 +-
 .../authorization_update_own_table.q|   2 +-
 .../clientpositive/auto_smb_mapjoin_14.q|   4 +-
 .../clientpositive/auto_sortmerge_join_10.q |   4 +-
 .../clientpositive/auto_sortmerge_join_13.q |   4 +-
 .../clientpositive/auto_sortmerge_join_14.q |   4 +-
 .../clientpositive/auto_sortmerge_join_15.q |   4 +-
 .../clientpositive/auto_sortmerge_join_16.q |   4 +-
 .../clientpositive/auto_sortmerge_join_6.q  |   4 +-
 .../clientpositive/auto_sortmerge_join_9.q  |   4 +-
 ql/src/test/queries/clientpositive/bucket1.q|   2 +-
 ql/src/test/queries/clientpositive/bucket2.q|   2 +-
 ql/src/test/queries/clientpositive/bucket3.q|   2 +-
 ql/src/test/queries/clientpositive/bucket4.q|   4 +-
 ql/src/test/queries/clientpositive/bucket5.q|   4 +-
 ql/src/test/queries/clientpositive/bucket6.q|   4 +-
 .../test/queries/clientpositive/bucket_many.q   |   2 +-
 .../queries/clientpositive/bucket_map_join_1.q  |   4 +-
 .../queries/clientpositive/bucket_map_join_2.q  |   4 +-
 .../clientpositive/bucket_map_join_spark4.q |   8 +-
 .../clientpositive/bucket_map_join_tez1.q   |   4 +-
 .../clientpositive/bucket_map_join_tez2.q   |   4 +-
 .../clientpositive/bucket_num_reducers.q|   2 +-
 .../clientpositive/bucket_num_reducers2.q   |   2 +-
 .../queries/clientpositive/bucketmapjoin13.q|   4 +-
 .../queries/clientpositive/bucketmapjoin6.q |   4 +-
 .../bucketsortoptimize_insert_1.q   |   4 +-
 .../bucketsortoptimize_insert_2.q   |   4 +-
 .../bucketsortoptimize_insert_3.q   |   4 +-
 .../bucketsortoptimize_insert_4.q   |   4 +-
 .../bucketsortoptimize_insert_5.q   |   4 +-
 .../bucketsortoptimize_insert_6.q   |   4 +-
 .../bucketsortoptimize_insert_7.q   |   4 +-
 .../bucketsortoptimize_insert_8.q   |   4 +-
 .../queries/clientpositive/cbo_rp_auto_join1.q  |   

[3/4] hive git commit: HIVE-12331 : Remove hive.enforce.bucketing & hive.enforce.sorting configs (Ashutosh Chauhan via Jason Dere)

[hashutosh]

http://git-wip-us.apache.org/repos/asf/hive/blob/5562fae7/ql/src/test/queries/clientpositive/delete_all_non_partitioned.q
--
diff --git a/ql/src/test/queries/clientpositive/delete_all_non_partitioned.q 
b/ql/src/test/queries/clientpositive/delete_all_non_partitioned.q
index 9110dcc..82c18e2 100644
--- a/ql/src/test/queries/clientpositive/delete_all_non_partitioned.q
+++ b/ql/src/test/queries/clientpositive/delete_all_non_partitioned.q
@@ -1,6 +1,6 @@
 set hive.support.concurrency=true;
 set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
-set hive.enforce.bucketing=true;
+
 
 create table acid_danp(a int, b varchar(128)) clustered by (a) into 2 buckets 
stored as orc TBLPROPERTIES ('transactional'='true');
 

http://git-wip-us.apache.org/repos/asf/hive/blob/5562fae7/ql/src/test/queries/clientpositive/delete_all_partitioned.q
--
diff --git a/ql/src/test/queries/clientpositive/delete_all_partitioned.q 
b/ql/src/test/queries/clientpositive/delete_all_partitioned.q
index f082b6d..122b3e2 100644
--- a/ql/src/test/queries/clientpositive/delete_all_partitioned.q
+++ b/ql/src/test/queries/clientpositive/delete_all_partitioned.q
@@ -1,6 +1,6 @@
 set hive.support.concurrency=true;
 set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
-set hive.enforce.bucketing=true;
+
 
 create table acid_dap(a int, b varchar(128)) partitioned by (ds string) 
clustered by (a) into 2 buckets stored as orc TBLPROPERTIES 
('transactional'='true');
 

http://git-wip-us.apache.org/repos/asf/hive/blob/5562fae7/ql/src/test/queries/clientpositive/delete_orig_table.q
--
diff --git a/ql/src/test/queries/clientpositive/delete_orig_table.q 
b/ql/src/test/queries/clientpositive/delete_orig_table.q
index fd23f4b..88cc830 100644
--- a/ql/src/test/queries/clientpositive/delete_orig_table.q
+++ b/ql/src/test/queries/clientpositive/delete_orig_table.q
@@ -1,6 +1,6 @@
 set hive.support.concurrency=true;
 set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
-set hive.enforce.bucketing=true;
+
 
 dfs ${system:test.dfs.mkdir} ${system:test.tmp.dir}/delete_orig_table;
 dfs -copyFromLocal ../../data/files/alltypesorc 
${system:test.tmp.dir}/delete_orig_table/0_0; 

http://git-wip-us.apache.org/repos/asf/hive/blob/5562fae7/ql/src/test/queries/clientpositive/delete_tmp_table.q
--
diff --git a/ql/src/test/queries/clientpositive/delete_tmp_table.q 
b/ql/src/test/queries/clientpositive/delete_tmp_table.q
index eb6c095..c7d8aa6 100644
--- a/ql/src/test/queries/clientpositive/delete_tmp_table.q
+++ b/ql/src/test/queries/clientpositive/delete_tmp_table.q
@@ -1,6 +1,6 @@
 set hive.support.concurrency=true;
 set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
-set hive.enforce.bucketing=true;
+
 
 create temporary table acid_dtt(a int, b varchar(128)) clustered by (a) into 2 
buckets stored as orc TBLPROPERTIES ('transactional'='true');
 

http://git-wip-us.apache.org/repos/asf/hive/blob/5562fae7/ql/src/test/queries/clientpositive/delete_where_no_match.q
--
diff --git a/ql/src/test/queries/clientpositive/delete_where_no_match.q 
b/ql/src/test/queries/clientpositive/delete_where_no_match.q
index 8ed979d..f13dd73 100644
--- a/ql/src/test/queries/clientpositive/delete_where_no_match.q
+++ b/ql/src/test/queries/clientpositive/delete_where_no_match.q
@@ -1,6 +1,6 @@
 set hive.support.concurrency=true;
 set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
-set hive.enforce.bucketing=true;
+
 
 create table acid_dwnm(a int, b varchar(128)) clustered by (a) into 2 buckets 
stored as orc TBLPROPERTIES ('transactional'='true');
 

http://git-wip-us.apache.org/repos/asf/hive/blob/5562fae7/ql/src/test/queries/clientpositive/delete_where_non_partitioned.q
--
diff --git a/ql/src/test/queries/clientpositive/delete_where_non_partitioned.q 
b/ql/src/test/queries/clientpositive/delete_where_non_partitioned.q
index dac5375..de1ca36 100644
--- a/ql/src/test/queries/clientpositive/delete_where_non_partitioned.q
+++ b/ql/src/test/queries/clientpositive/delete_where_non_partitioned.q
@@ -1,6 +1,6 @@
 set hive.support.concurrency=true;
 set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
-set hive.enforce.bucketing=true;
+
 
 create table acid_dwnp(a int, b varchar(128)) clustered by (a) into 2 buckets 
stored as orc TBLPROPERTIES ('transactional'='true');
 

http://git-wip-us.apache.org/repos/asf/hive/blob/5562fae7/ql/src/test/queries/clientpositive/delete_where_partitioned.q
--
diff --git 

[1/3] hive git commit: HIVE-12329 :Turn on limit pushdown optimization by default (Ashutosh Chauhan via Prasanth J)

[hashutosh]

Repository: hive
Updated Branches:
  refs/heads/master 5562fae73 -> daf46aa76


http://git-wip-us.apache.org/repos/asf/hive/blob/daf46aa7/ql/src/test/results/clientpositive/tez/vector_cast_constant.q.java1.7.out
--
diff --git 
a/ql/src/test/results/clientpositive/tez/vector_cast_constant.q.java1.7.out 
b/ql/src/test/results/clientpositive/tez/vector_cast_constant.q.java1.7.out
index 12920d2..bcbdf06 100644
--- a/ql/src/test/results/clientpositive/tez/vector_cast_constant.q.java1.7.out
+++ b/ql/src/test/results/clientpositive/tez/vector_cast_constant.q.java1.7.out
@@ -161,6 +161,7 @@ STAGE PLANS:
   key expressions: _col0 (type: int)
   sort order: +
   Statistics: Num rows: 524 Data size: 155436 Basic stats: 
COMPLETE Column stats: NONE
+  TopN Hash Memory Usage: 0.1
   value expressions: _col1 (type: double), _col2 (type: 
double), _col3 (type: decimal(14,4))
 Reducer 3 
 Execution mode: vectorized

http://git-wip-us.apache.org/repos/asf/hive/blob/daf46aa7/ql/src/test/results/clientpositive/tez/vector_char_2.q.out
--
diff --git a/ql/src/test/results/clientpositive/tez/vector_char_2.q.out 
b/ql/src/test/results/clientpositive/tez/vector_char_2.q.out
index 8545608..f88ee91 100644
--- a/ql/src/test/results/clientpositive/tez/vector_char_2.q.out
+++ b/ql/src/test/results/clientpositive/tez/vector_char_2.q.out
@@ -106,6 +106,7 @@ STAGE PLANS:
   key expressions: _col0 (type: char(20))
   sort order: +
   Statistics: Num rows: 250 Data size: 49500 Basic stats: 
COMPLETE Column stats: NONE
+  TopN Hash Memory Usage: 0.1
   value expressions: _col1 (type: bigint), _col2 (type: bigint)
 Reducer 3 
 Execution mode: vectorized
@@ -234,6 +235,7 @@ STAGE PLANS:
   key expressions: _col0 (type: char(20))
   sort order: -
   Statistics: Num rows: 250 Data size: 49500 Basic stats: 
COMPLETE Column stats: NONE
+  TopN Hash Memory Usage: 0.1
   value expressions: _col1 (type: bigint), _col2 (type: bigint)
 Reducer 3 
 Execution mode: vectorized

http://git-wip-us.apache.org/repos/asf/hive/blob/daf46aa7/ql/src/test/results/clientpositive/tez/vector_char_simple.q.out
--
diff --git a/ql/src/test/results/clientpositive/tez/vector_char_simple.q.out 
b/ql/src/test/results/clientpositive/tez/vector_char_simple.q.out
index be38775..617620c 100644
--- a/ql/src/test/results/clientpositive/tez/vector_char_simple.q.out
+++ b/ql/src/test/results/clientpositive/tez/vector_char_simple.q.out
@@ -79,6 +79,7 @@ STAGE PLANS:
   key expressions: _col0 (type: char(10))
   sort order: +
   Statistics: Num rows: 500 Data size: 99000 Basic stats: 
COMPLETE Column stats: NONE
+  TopN Hash Memory Usage: 0.1
   value expressions: _col1 (type: char(20))
 Execution mode: vectorized
 Reducer 2 
@@ -179,6 +180,7 @@ STAGE PLANS:
   key expressions: _col0 (type: char(10))
   sort order: -
   Statistics: Num rows: 500 Data size: 99000 Basic stats: 
COMPLETE Column stats: NONE
+  TopN Hash Memory Usage: 0.1
   value expressions: _col1 (type: char(20))
 Execution mode: vectorized
 Reducer 2 
@@ -282,6 +284,7 @@ STAGE PLANS:
   Reduce Output Operator
 sort order: 
 Statistics: Num rows: 10 Data size: 2150 Basic stats: 
COMPLETE Column stats: NONE
+TopN Hash Memory Usage: 0.1
 value expressions: _col0 (type: int)
 Execution mode: vectorized
 Reducer 2 

http://git-wip-us.apache.org/repos/asf/hive/blob/daf46aa7/ql/src/test/results/clientpositive/tez/vector_coalesce.q.out
--
diff --git a/ql/src/test/results/clientpositive/tez/vector_coalesce.q.out 
b/ql/src/test/results/clientpositive/tez/vector_coalesce.q.out
index c492113..1142485 100644
--- a/ql/src/test/results/clientpositive/tez/vector_coalesce.q.out
+++ b/ql/src/test/results/clientpositive/tez/vector_coalesce.q.out
@@ -41,6 +41,7 @@ STAGE PLANS:
 key expressions: null (type: double), _col1 (type: 
string), _col2 (type: int), _col3 (type: float), _col4 (type: smallint), _col5 
(type: string)
 sort order: ++
 Statistics: Num rows: 6144 Data size: 1320982 Basic 
stats: COMPLETE Column stats: NONE
+   

[3/3] hive git commit: HIVE-12329 :Turn on limit pushdown optimization by default (Ashutosh Chauhan via Prasanth J)

[hashutosh]

HIVE-12329 :Turn on limit pushdown optimization by default (Ashutosh Chauhan 
via Prasanth J)

Signed-off-by: Ashutosh Chauhan 


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/daf46aa7
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/daf46aa7
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/daf46aa7

Branch: refs/heads/master
Commit: daf46aa768bdb6745fe1509bcf98aad537be367b
Parents: 5562fae
Author: Ashutosh Chauhan 
Authored: Tue Nov 3 17:26:00 2015 -0800
Committer: Ashutosh Chauhan 
Committed: Tue Nov 24 18:11:41 2015 -0800

--
 .../org/apache/hadoop/hive/conf/HiveConf.java   |  4 +--
 .../clientpositive/annotate_stats_select.q.out  |  4 +++
 .../auto_join_without_localtask.q.out   |  3 +++
 .../results/clientpositive/bucket_groupby.q.out | 10 
 .../results/clientpositive/bucketmapjoin7.q.out |  2 ++
 ql/src/test/results/clientpositive/cp_sel.q.out |  3 ++-
 ql/src/test/results/clientpositive/ctas.q.out   | 12 +
 .../results/clientpositive/ctas_colname.q.out   |  2 ++
 .../dynpart_sort_opt_vectorization.q.out|  3 +++
 .../dynpart_sort_optimization.q.out |  3 +++
 .../encryption_insert_partition_dynamic.q.out   |  2 ++
 .../test/results/clientpositive/gby_star.q.out  |  4 +++
 .../results/clientpositive/groupby1_limit.q.out |  2 ++
 .../results/clientpositive/groupby2_limit.q.out |  1 +
 .../groupby7_noskew_multi_single_reducer.q.out  |  2 ++
 ...pby_complex_types_multi_single_reducer.q.out |  2 ++
 .../groupby_multi_single_reducer.q.out  |  1 +
 .../results/clientpositive/input11_limit.q.out  |  1 +
 .../results/clientpositive/input14_limit.q.out  |  2 ++
 .../results/clientpositive/input1_limit.q.out   |  2 ++
 .../test/results/clientpositive/input22.q.out   |  1 +
 .../test/results/clientpositive/input25.q.out   |  2 ++
 .../test/results/clientpositive/input26.q.out   |  2 ++
 .../results/clientpositive/input3_limit.q.out   |  2 ++
 .../results/clientpositive/input4_limit.q.out   |  2 ++
 .../results/clientpositive/input_part10.q.out   |  1 +
 .../insert1_overwrite_partitions.q.out  |  2 ++
 .../insert2_overwrite_partitions.q.out  |  2 ++
 .../results/clientpositive/insert_into1.q.out   |  3 +++
 .../results/clientpositive/insert_into2.q.out   |  3 +++
 .../results/clientpositive/insert_into3.q.out   |  4 +++
 .../results/clientpositive/insert_into4.q.out   |  2 ++
 .../results/clientpositive/insert_into5.q.out   |  1 +
 .../results/clientpositive/insert_into6.q.out   |  1 +
 .../test/results/clientpositive/join_vc.q.out   |  1 +
 .../results/clientpositive/lateral_view.q.out   |  3 +++
 .../clientpositive/lateral_view_explode2.q.out  |  2 ++
 .../clientpositive/lateral_view_noalias.q.out   |  4 +++
 .../clientpositive/lateral_view_onview.q.out|  3 +++
 .../clientpositive/load_dyn_part14.q.out|  3 +++
 ql/src/test/results/clientpositive/merge4.q.out |  2 ++
 .../nonreserved_keywords_insert_into1.q.out |  3 +++
 .../results/clientpositive/orc_createas1.q.out  |  1 +
 .../clientpositive/orc_predicate_pushdown.q.out |  6 +
 ql/src/test/results/clientpositive/order.q.out  |  2 ++
 ql/src/test/results/clientpositive/order2.q.out |  1 +
 .../parquet_predicate_pushdown.q.out|  8 ++
 ql/src/test/results/clientpositive/pcr.q.out|  2 ++
 .../test/results/clientpositive/regex_col.q.out |  1 +
 .../results/clientpositive/script_pipe.q.out|  1 +
 .../clientpositive/select_as_omitted.q.out  |  1 +
 .../clientpositive/skewjoin_noskew.q.out|  1 +
 .../results/clientpositive/smb_mapjoin_13.q.out |  4 +++
 .../results/clientpositive/smb_mapjoin_15.q.out |  8 ++
 .../spark/auto_join_without_localtask.q.out |  3 +++
 .../clientpositive/spark/bucketmapjoin7.q.out   |  2 ++
 .../results/clientpositive/spark/ctas.q.out | 12 +
 .../groupby7_noskew_multi_single_reducer.q.out  |  1 +
 ...pby_complex_types_multi_single_reducer.q.out |  2 ++
 .../spark/groupby_multi_single_reducer.q.out|  1 +
 .../clientpositive/spark/input1_limit.q.out |  2 ++
 .../clientpositive/spark/insert_into1.q.out |  3 +++
 .../clientpositive/spark/insert_into2.q.out |  3 +++
 .../clientpositive/spark/insert_into3.q.out | 18 -
 .../results/clientpositive/spark/join_vc.q.out  |  1 +
 .../spark/lateral_view_explode2.q.out   |  2 ++
 .../clientpositive/spark/load_dyn_part14.q.out  |  1 +
 .../results/clientpositive/spark/order.q.out|  2 ++
 .../results/clientpositive/spark/order2.q.out   |  1 +
 .../test/results/clientpositive/spark/pcr.q.out |  2 ++
 .../clientpositive/spark/script_pipe.q.out  |  1 +
 .../clientpositive/spark/skewjoin_noskew.q.out  |  1 +
 .../clientpositive/spark/smb_mapjoin_13.q.out   |  4 +++
 

[2/3] hive git commit: HIVE-12329 :Turn on limit pushdown optimization by default (Ashutosh Chauhan via Prasanth J)

[hashutosh]

http://git-wip-us.apache.org/repos/asf/hive/blob/daf46aa7/ql/src/test/results/clientpositive/smb_mapjoin_15.q.out
--
diff --git a/ql/src/test/results/clientpositive/smb_mapjoin_15.q.out 
b/ql/src/test/results/clientpositive/smb_mapjoin_15.q.out
index 64a3ea2..8608187 100644
--- a/ql/src/test/results/clientpositive/smb_mapjoin_15.q.out
+++ b/ql/src/test/results/clientpositive/smb_mapjoin_15.q.out
@@ -124,6 +124,8 @@ STAGE PLANS:
 key expressions: _col0 (type: int)
 sort order: +
 tag: -1
+TopN: 10
+TopN Hash Memory Usage: 0.1
 value expressions: _col1 (type: string), _col2 (type: 
int), _col3 (type: string)
 auto parallelism: false
   Path -> Alias:
@@ -382,6 +384,8 @@ STAGE PLANS:
 key expressions: _col0 (type: int)
 sort order: +
 tag: -1
+TopN: 10
+TopN Hash Memory Usage: 0.1
 value expressions: _col1 (type: int), _col2 (type: 
string), _col3 (type: int), _col4 (type: int), _col5 (type: string)
 auto parallelism: false
   Path -> Alias:
@@ -588,6 +592,8 @@ STAGE PLANS:
 key expressions: _col0 (type: int)
 sort order: +
 tag: -1
+TopN: 10
+TopN Hash Memory Usage: 0.1
 value expressions: _col1 (type: int), _col2 (type: 
string), _col3 (type: int), _col4 (type: int), _col5 (type: string)
 auto parallelism: false
   Path -> Alias:
@@ -827,6 +833,8 @@ STAGE PLANS:
 sort order: +
 Statistics: Num rows: 137 Data size: 1984 Basic stats: 
COMPLETE Column stats: NONE
 tag: -1
+TopN: 10
+TopN Hash Memory Usage: 0.1
 value expressions: _col1 (type: int), _col2 (type: 
string), _col3 (type: int), _col4 (type: int), _col5 (type: string)
 auto parallelism: false
   Local Work:

http://git-wip-us.apache.org/repos/asf/hive/blob/daf46aa7/ql/src/test/results/clientpositive/spark/auto_join_without_localtask.q.out
--
diff --git 
a/ql/src/test/results/clientpositive/spark/auto_join_without_localtask.q.out 
b/ql/src/test/results/clientpositive/spark/auto_join_without_localtask.q.out
index a234ff5..932fdcc 100644
--- a/ql/src/test/results/clientpositive/spark/auto_join_without_localtask.q.out
+++ b/ql/src/test/results/clientpositive/spark/auto_join_without_localtask.q.out
@@ -65,6 +65,7 @@ STAGE PLANS:
   key expressions: _col0 (type: string), _col1 (type: string)
   sort order: ++
   Statistics: Num rows: 275 Data size: 2921 Basic stats: 
COMPLETE Column stats: NONE
+  TopN Hash Memory Usage: 0.1
 Reducer 3 
 Reduce Operator Tree:
   Select Operator
@@ -238,6 +239,7 @@ STAGE PLANS:
   key expressions: _col0 (type: string), _col1 (type: string)
   sort order: ++
   Statistics: Num rows: 302 Data size: 3213 Basic stats: 
COMPLETE Column stats: NONE
+  TopN Hash Memory Usage: 0.1
 Reducer 4 
 Reduce Operator Tree:
   Select Operator
@@ -411,6 +413,7 @@ STAGE PLANS:
   key expressions: _col0 (type: string), _col1 (type: string)
   sort order: ++
   Statistics: Num rows: 275 Data size: 2921 Basic stats: 
COMPLETE Column stats: NONE
+  TopN Hash Memory Usage: 0.1
 Reducer 4 
 Reduce Operator Tree:
   Select Operator

http://git-wip-us.apache.org/repos/asf/hive/blob/daf46aa7/ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out
--
diff --git a/ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out 
b/ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out
index a99cb74..84f68a3 100644
--- a/ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out
+++ b/ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out
@@ -262,6 +262,8 @@ STAGE PLANS:
   sort order: ++
   Statistics: Num rows: 378 Data size: 1514 Basic 
stats: COMPLETE Column stats: NONE
   tag: -1
+  TopN: 1
+  TopN Hash Memory Usage: 0.1
   auto parallelism: false
 Local Work:
   Map Reduce Local Work

http://git-wip-us.apache.org/repos/asf/hive/blob/daf46aa7/ql/src/test/results/clientpositive/spark/ctas.q.out

[1/4] hive git commit: HIVE-12331 : Remove hive.enforce.bucketing & hive.enforce.sorting configs (Ashutosh Chauhan via Jason Dere)

[hashutosh]

Repository: hive
Updated Branches:
  refs/heads/master 1b6600de0 -> 5562fae73


http://git-wip-us.apache.org/repos/asf/hive/blob/5562fae7/ql/src/test/results/clientpositive/spark/smb_mapjoin_12.q.out
--
diff --git a/ql/src/test/results/clientpositive/spark/smb_mapjoin_12.q.out 
b/ql/src/test/results/clientpositive/spark/smb_mapjoin_12.q.out
index eeb18b0..93a7ca4 100644
--- a/ql/src/test/results/clientpositive/spark/smb_mapjoin_12.q.out
+++ b/ql/src/test/results/clientpositive/spark/smb_mapjoin_12.q.out
@@ -139,6 +139,8 @@ STAGE DEPENDENCIES:
 STAGE PLANS:
   Stage: Stage-1
 Spark
+  Edges:
+Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1)
  A masked pattern was here 
   Vertices:
 Map 1 
@@ -165,37 +167,14 @@ STAGE PLANS:
 expressions: _col0 (type: int), _col7 (type: string)
 outputColumnNames: _col0, _col1
 Statistics: Num rows: 825 Data size: 8764 Basic stats: 
COMPLETE Column stats: NONE
-File Output Operator
-  compressed: false
-  GlobalTableId: 1
- A masked pattern was here 
-  NumFilesPerFileSink: 1
-  Static Partition Specification: ds=1/
+Reduce Output Operator
+  key expressions: _col0 (type: int)
+  sort order: +
+  Map-reduce partition columns: _col0 (type: int)
   Statistics: Num rows: 825 Data size: 8764 Basic 
stats: COMPLETE Column stats: NONE
- A masked pattern was here 
-  table:
-  input format: 
org.apache.hadoop.mapred.TextInputFormat
-  output format: 
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
-  properties:
-SORTBUCKETCOLSPREFIX TRUE
-bucket_count 16
-bucket_field_name key
-columns key,value
-columns.comments 
-columns.types int:string
- A masked pattern was here 
-name default.test_table3
-partition_columns ds
-partition_columns.types string
-serialization.ddl struct test_table3 { i32 
key, string value}
-serialization.format 1
-serialization.lib 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- A masked pattern was here 
-  serde: 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
-  name: default.test_table3
-  TotalFiles: 1
-  GatherStats: true
-  MultiFileSpray: false
+  tag: -1
+  value expressions: _col1 (type: string)
+  auto parallelism: false
 Path -> Alias:
  A masked pattern was here 
 Path -> Partition:
@@ -249,6 +228,44 @@ STAGE PLANS:
   name: default.test_table1
 Truncated Path -> Alias:
   /test_table1/ds=1 [a]
+Reducer 2 
+Needs Tagging: false
+Reduce Operator Tree:
+  Select Operator
+expressions: KEY.reducesinkkey0 (type: int), VALUE._col0 
(type: string)
+outputColumnNames: _col0, _col1
+Statistics: Num rows: 825 Data size: 8764 Basic stats: 
COMPLETE Column stats: NONE
+File Output Operator
+  compressed: false
+  GlobalTableId: 1
+ A masked pattern was here 
+  NumFilesPerFileSink: 16
+  Static Partition Specification: ds=1/
+  Statistics: Num rows: 825 Data size: 8764 Basic stats: 
COMPLETE Column stats: NONE
+ A masked pattern was here 
+  table:
+  input format: org.apache.hadoop.mapred.TextInputFormat
+  output format: 
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+  properties:
+SORTBUCKETCOLSPREFIX TRUE
+bucket_count 16
+bucket_field_name key
+columns key,value
+columns.comments 
+columns.types int:string
+ A masked pattern was here 
+name default.test_table3
+partition_columns ds
+partition_columns.types string
+

hive git commit: HIVE-12466: SparkCounter not initialized error (Rui via Chengxiang)

[chengxiang]

Repository: hive
Updated Branches:
  refs/heads/spark b934ad1cc -> eddb8ca1d


HIVE-12466: SparkCounter not initialized error (Rui via Chengxiang)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/eddb8ca1
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/eddb8ca1
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/eddb8ca1

Branch: refs/heads/spark
Commit: eddb8ca1d345f5a18f0b75895c92620e69956f6a
Parents: b934ad1
Author: chengxiang 
Authored: Wed Nov 25 11:07:12 2015 +0800
Committer: chengxiang 
Committed: Wed Nov 25 11:07:12 2015 +0800

--
 .../hadoop/hive/ql/exec/FileSinkOperator.java  | 17 ++---
 .../hadoop/hive/ql/exec/ReduceSinkOperator.java| 14 +-
 .../hadoop/hive/ql/exec/spark/SparkTask.java   |  4 ++--
 3 files changed, 21 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/eddb8ca1/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
--
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
index 7459bba..99cdb95 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
@@ -430,13 +430,7 @@ public class FileSinkOperator extends 
TerminalOperator implements
   cntr = 1;
   logEveryNRows = HiveConf.getLongVar(hconf, 
HiveConf.ConfVars.HIVE_LOG_N_RECORDS);
 
-  String suffix = Integer.toString(conf.getDestTableId());
-  String fullName = conf.getTableInfo().getTableName();
-  if (fullName != null) {
-suffix = suffix + "_" + fullName.toLowerCase();
-  }
-
-  statsMap.put(Counter.RECORDS_OUT + "_" + suffix, row_count);
+  statsMap.put(getCounterName(Counter.RECORDS_OUT), row_count);
 } catch (HiveException e) {
   throw e;
 } catch (Exception e) {
@@ -445,6 +439,15 @@ public class FileSinkOperator extends 
TerminalOperator implements
 }
   }
 
+  public String getCounterName(Counter counter) {
+String suffix = Integer.toString(conf.getDestTableId());
+String fullName = conf.getTableInfo().getTableName();
+if (fullName != null) {
+  suffix = suffix + "_" + fullName.toLowerCase();
+}
+return counter + "_" + suffix;
+  }
+
   private void logOutputFormatError(Configuration hconf, HiveException ex) {
 StringWriter errorWriter = new StringWriter();
 errorWriter.append("Failed to create output format; configuration: ");

http://git-wip-us.apache.org/repos/asf/hive/blob/eddb8ca1/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
--
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
index ef5ee95..f4730ec 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
@@ -164,11 +164,7 @@ public class ReduceSinkOperator extends 
TerminalOperator
   cntr = 1;
   logEveryNRows = HiveConf.getLongVar(hconf, 
HiveConf.ConfVars.HIVE_LOG_N_RECORDS);
 
-  String context = hconf.get(Operator.CONTEXT_NAME_KEY, "");
-  if (context != null && !context.isEmpty()) {
-context = "_" + context.replace(" ","_");
-  }
-  statsMap.put(Counter.RECORDS_OUT_INTERMEDIATE + context, recordCounter);
+  statsMap.put(getCounterName(Counter.RECORDS_OUT_INTERMEDIATE, hconf), 
recordCounter);
 
   List keys = conf.getKeyCols();
 
@@ -250,6 +246,14 @@ public class ReduceSinkOperator extends 
TerminalOperator
 }
   }
 
+  public String getCounterName(Counter counter, Configuration hconf) {
+String context = hconf.get(Operator.CONTEXT_NAME_KEY, "");
+if (context != null && !context.isEmpty()) {
+  context = "_" + context.replace(" ", "_");
+}
+return counter + context;
+  }
+
 
   /**
* Initializes array of ExprNodeEvaluator. Adds Union field for distinct

http://git-wip-us.apache.org/repos/asf/hive/blob/eddb8ca1/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java
--
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java
index 336d490..eaeffee 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java
@@ -384,11 +384,11 @@ public class SparkTask extends Task {
   for (Operator operator : 

hive git commit: HIVE-12175: Upgrade Kryo version to 3.0.x (Prasanth Jayachandran reviewed by Ashutosh Chauhan)

[prasanthj]

Repository: hive
Updated Branches:
  refs/heads/master c6a835c43 -> b7281ce6a


HIVE-12175: Upgrade Kryo version to 3.0.x (Prasanth Jayachandran reviewed by 
Ashutosh Chauhan)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/b7281ce6
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/b7281ce6
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/b7281ce6

Branch: refs/heads/master
Commit: b7281ce6a61bcfbd398fd691cddc38c4f1a61f64
Parents: c6a835c
Author: Prasanth Jayachandran 
Authored: Tue Nov 24 12:43:46 2015 -0600
Committer: Prasanth Jayachandran 
Committed: Tue Nov 24 12:43:46 2015 -0600

--
 itests/qtest-accumulo/pom.xml   |   2 +-
 pom.xml |   6 +-
 ql/pom.xml  |  36 +++--
 .../apache/hadoop/hive/ql/exec/Utilities.java   | 145 +--
 .../org/apache/hadoop/hive/ql/plan/MapWork.java |  15 --
 .../apache/hadoop/hive/ql/plan/ReduceWork.java  |   5 -
 spark-client/pom.xml|  28 ++--
 .../hive/spark/client/rpc/KryoMessageCodec.java |  11 +-
 8 files changed, 185 insertions(+), 63 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/b7281ce6/itests/qtest-accumulo/pom.xml
--
diff --git a/itests/qtest-accumulo/pom.xml b/itests/qtest-accumulo/pom.xml
index 7403a15..f7325dc 100644
--- a/itests/qtest-accumulo/pom.xml
+++ b/itests/qtest-accumulo/pom.xml
@@ -123,7 +123,7 @@
 
 
-  com.esotericsoftware.kryo
+  com.esotericsoftware
   kryo
   ${kryo.version}
   test

http://git-wip-us.apache.org/repos/asf/hive/blob/b7281ce6/pom.xml
--
diff --git a/pom.xml b/pom.xml
index c6df4a5..c38c10f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -144,7 +144,7 @@
 3.5.2
 20090211
 4.11
-2.22
+3.0.3
 0.9.3
 0.9.3
 2.4
@@ -228,8 +228,8 @@
 
   
   
-com.esotericsoftware.kryo
-kryo
+com.esotericsoftware
+kryo-shaded
 ${kryo.version}
   
   

http://git-wip-us.apache.org/repos/asf/hive/blob/b7281ce6/ql/pom.xml
--
diff --git a/ql/pom.xml b/ql/pom.xml
index 9420a62..d893099 100644
--- a/ql/pom.xml
+++ b/ql/pom.xml
@@ -72,8 +72,8 @@
 
 
 
-  com.esotericsoftware.kryo
-  kryo
+  com.esotericsoftware
+  kryo-shaded
   ${kryo.version}
 
 
@@ -594,16 +594,20 @@
   spark-core_${scala.binary.version}
   ${spark.version}
   true
-
- 
-org.slf4j
-slf4j-log4j12
-  
-  
-commmons-logging
-commons-logging
-  
-
+  
+   
+ com.esotericsoftware.kryo
+ kryo
+   
+   
+ org.slf4j
+ slf4j-log4j12
+   
+   
+ commmons-logging
+ commons-logging
+   
+ 

 
   com.sun.jersey
@@ -746,7 +750,9 @@
   org.apache.hive:hive-serde
   org.apache.hive:hive-llap-client
   org.apache.hive:hive-metastore
-  com.esotericsoftware.kryo:kryo
+  com.esotericsoftware:kryo-shaded
+ com.esotericsoftware:minlog
+ org.objenesis:objenesis
   org.apache.parquet:parquet-hadoop-bundle
   org.apache.thrift:libthrift
   org.apache.thrift:libfb303
@@ -779,6 +785,10 @@
   com.esotericsoftware
   
org.apache.hive.com.esotericsoftware
 
+
+  org.objenesis
+  org.apache.hive.org.objenesis
+
   
 
   

http://git-wip-us.apache.org/repos/asf/hive/blob/b7281ce6/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
--
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
index 9dbb45a..8b8cf6d 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
@@ -37,6 +37,8 @@ import java.io.InputStream;
 import java.io.OutputStream;
 import java.io.Serializable;
 import java.io.UnsupportedEncodingException;
+import java.lang.reflect.Array;
+import java.lang.reflect.Field;
 import java.net.URI;
 import java.net.URL;
 import java.net.URLClassLoader;
@@ -87,8 +89,6 @@ import org.apache.commons.lang.StringUtils;
 import 

hive git commit: HIVE-12509: Regenerate q files after HIVE-12017 went in (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)

[jcamacho]

Repository: hive
Updated Branches:
  refs/heads/master ab98ffc26 -> c6a835c43


HIVE-12509: Regenerate q files after HIVE-12017 went in (Jesus Camacho 
Rodriguez, reviewed by Ashutosh Chauhan)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/c6a835c4
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/c6a835c4
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/c6a835c4

Branch: refs/heads/master
Commit: c6a835c432ac161611ee75eab3cbb34cda414de3
Parents: ab98ffc
Author: Jesus Camacho Rodriguez 
Authored: Tue Nov 24 17:08:20 2015 +0100
Committer: Jesus Camacho Rodriguez 
Committed: Tue Nov 24 17:14:48 2015 +0100

--
 .../results/clientpositive/subquery_notin.q.out | 328 ++-
 .../subquery_notin_having.q.java1.7.out |  94 +++---
 .../clientpositive/tez/explainuser_1.q.out  | 171 +-
 .../tez/tez_dynpart_hashjoin_3.q.out| 150 -
 .../clientpositive/tez/tez_smb_empty.q.out  |  38 ++-
 5 files changed, 398 insertions(+), 383 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/c6a835c4/ql/src/test/results/clientpositive/subquery_notin.q.out
--
diff --git a/ql/src/test/results/clientpositive/subquery_notin.q.out 
b/ql/src/test/results/clientpositive/subquery_notin.q.out
index 552999d..56553fd 100644
--- a/ql/src/test/results/clientpositive/subquery_notin.q.out
+++ b/ql/src/test/results/clientpositive/subquery_notin.q.out
@@ -1,4 +1,4 @@
-Warning: Shuffle Join JOIN[23][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 
'Stage-2:MAPRED' is a cross product
+Warning: Shuffle Join JOIN[20][tables = [$hdt$_0, $hdt$_1]] in Stage 
'Stage-1:MAPRED' is a cross product
 PREHOOK: query: -- non agg, non corr
 explain
 select * 
@@ -18,70 +18,64 @@ where src.key not in
   )
 POSTHOOK: type: QUERY
 STAGE DEPENDENCIES:
-  Stage-1 is a root stage
-  Stage-2 depends on stages: Stage-1, Stage-4
-  Stage-4 is a root stage
+  Stage-3 is a root stage
+  Stage-1 depends on stages: Stage-3
+  Stage-2 depends on stages: Stage-1
   Stage-0 depends on stages: Stage-2
 
 STAGE PLANS:
-  Stage: Stage-1
+  Stage: Stage-3
 Map Reduce
   Map Operator Tree:
   TableScan
 alias: src
 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE 
Column stats: NONE
-Select Operator
-  expressions: key (type: string), value (type: string)
-  outputColumnNames: _col0, _col1
-  Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE 
Column stats: NONE
-  Reduce Output Operator
-key expressions: _col0 (type: string)
-sort order: +
-Map-reduce partition columns: _col0 (type: string)
-Statistics: Num rows: 500 Data size: 5312 Basic stats: 
COMPLETE Column stats: NONE
-value expressions: _col1 (type: string)
-  TableScan
-alias: src
-Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE 
Column stats: NONE
 Filter Operator
-  predicate: (key > '2') (type: boolean)
-  Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE 
Column stats: NONE
+  predicate: ((key > '2') and key is null) (type: boolean)
+  Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE 
Column stats: NONE
   Select Operator
-expressions: key (type: string)
-outputColumnNames: _col0
-Statistics: Num rows: 166 Data size: 1763 Basic stats: 
COMPLETE Column stats: NONE
-Reduce Output Operator
-  key expressions: _col0 (type: string)
-  sort order: +
-  Map-reduce partition columns: _col0 (type: string)
-  Statistics: Num rows: 166 Data size: 1763 Basic stats: 
COMPLETE Column stats: NONE
+Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE 
Column stats: NONE
+Group By Operator
+  aggregations: count()
+  mode: hash
+  outputColumnNames: _col0
+  Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE 
Column stats: NONE
+  Reduce Output Operator
+sort order: 
+Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE 
Column stats: NONE
+value expressions: _col0 (type: bigint)
   Reduce Operator Tree:
-Join Operator
-  condition map:
-   Left Outer Join0 to 1
-  keys:
-0 _col0 (type: string)
-1 _col0 (type: string)
-  

[2/2] hive git commit: HIVE-12411: Remove counter based stats collection mechanism (Pengcheng Xiong, reviewed by Ashutosh Chauhan)

[pxiong]

HIVE-12411: Remove counter based stats collection mechanism (Pengcheng Xiong, 
reviewed by Ashutosh Chauhan)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/ab98ffc2
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/ab98ffc2
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/ab98ffc2

Branch: refs/heads/master
Commit: ab98ffc2688abbc75de13524ca46848e566354ef
Parents: 306a640
Author: Pengcheng Xiong 
Authored: Tue Nov 24 17:09:40 2015 +0800
Committer: Pengcheng Xiong 
Committed: Tue Nov 24 17:09:40 2015 +0800

--
 .../hadoop/hive/common/StatsSetupConst.java |  13 -
 .../org/apache/hadoop/hive/conf/HiveConf.java   |   8 +-
 data/conf/llap/hive-site.xml|   2 +-
 data/conf/spark/standalone/hive-site.xml|   2 +-
 data/conf/spark/yarn-client/hive-site.xml   |   2 +-
 data/conf/tez/hive-site.xml |   4 +-
 .../hive/ql/stats/CounterStatsAggregator.java   |  82 
 .../ql/stats/CounterStatsAggregatorSpark.java   |  58 ---
 .../ql/stats/CounterStatsAggregatorTez.java |  79 
 .../hive/ql/stats/CounterStatsPublisher.java|  66 ---
 .../hadoop/hive/ql/stats/StatsFactory.java  |  11 -
 .../test/queries/clientpositive/index_bitmap3.q |   1 -
 .../queries/clientpositive/index_bitmap_auto.q  |   1 -
 .../test/queries/clientpositive/stats_counter.q |  16 -
 .../clientpositive/stats_counter_partitioned.q  |  45 --
 .../clientpositive/llap/stats_counter.q.out | 102 
 .../llap/stats_counter_partitioned.q.out| 465 ---
 .../clientpositive/spark/stats_counter.q.out| 102 
 .../spark/stats_counter_partitioned.q.out   | 465 ---
 .../results/clientpositive/stats_counter.q.out  | 102 
 .../stats_counter_partitioned.q.out | 465 ---
 .../clientpositive/tez/metadataonly1.q.out  |  72 +--
 .../clientpositive/tez/optimize_nullscan.q.out  |  90 ++--
 .../clientpositive/tez/stats_counter.q.out  | 102 
 .../tez/stats_counter_partitioned.q.out | 465 ---
 25 files changed, 88 insertions(+), 2732 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/ab98ffc2/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java
--
diff --git a/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 
b/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java
index 0a44bde..2ff76ee 100644
--- a/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java
+++ b/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java
@@ -31,19 +31,6 @@ import java.util.Map;
 public class StatsSetupConst {
 
   public enum StatDB {
-counter {
-  @Override
-  public String getPublisher(Configuration conf) {
-return "org.apache.hadoop.hive.ql.stats.CounterStatsPublisher"; }
-  @Override
-  public String getAggregator(Configuration conf) {
-if (HiveConf.getVar(conf, 
HiveConf.ConfVars.HIVE_EXECUTION_ENGINE).equals("tez")) {
-  return "org.apache.hadoop.hive.ql.stats.CounterStatsAggregatorTez";
-} else if (HiveConf.getVar(conf, 
HiveConf.ConfVars.HIVE_EXECUTION_ENGINE).equals("spark")) {
-  return "org.apache.hadoop.hive.ql.stats.CounterStatsAggregatorSpark";
-}
-return "org.apache.hadoop.hive.ql.stats.CounterStatsAggregator"; }
-},
 fs {
   @Override
   public String getPublisher(Configuration conf) {

http://git-wip-us.apache.org/repos/asf/hive/blob/ab98ffc2/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
--
diff --git a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
index f48403b..fffedd9 100644
--- a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
+++ b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
@@ -1354,10 +1354,10 @@ public class HiveConf extends Configuration {
 // Statistics
 HIVESTATSAUTOGATHER("hive.stats.autogather", true,
 "A flag to gather statistics automatically during the INSERT OVERWRITE 
command."),
-HIVESTATSDBCLASS("hive.stats.dbclass", "fs", new PatternSet("counter", 
"custom", "fs"),
+HIVESTATSDBCLASS("hive.stats.dbclass", "fs", new PatternSet("custom", 
"fs"),
 "The storage that stores temporary Hive statistics. In filesystem 
based statistics collection ('fs'), \n" +
 "each task writes statistics it has collected in a file on the 
filesystem, which will be aggregated \n" +
-"after the job has finished. Supported values are fs (filesystem), 
counter, and custom as defined in 

[1/2] hive git commit: HIVE-12411: Remove counter based stats collection mechanism (Pengcheng Xiong, reviewed by Ashutosh Chauhan)

[pxiong]

Repository: hive
Updated Branches:
  refs/heads/master 306a64024 -> ab98ffc26


http://git-wip-us.apache.org/repos/asf/hive/blob/ab98ffc2/ql/src/test/results/clientpositive/stats_counter_partitioned.q.out
--
diff --git a/ql/src/test/results/clientpositive/stats_counter_partitioned.q.out 
b/ql/src/test/results/clientpositive/stats_counter_partitioned.q.out
deleted file mode 100644
index 626dcff..000
--- a/ql/src/test/results/clientpositive/stats_counter_partitioned.q.out
+++ /dev/null
@@ -1,465 +0,0 @@
-PREHOOK: query: -- partitioned table analyze 
-
-create table dummy (key string, value string) partitioned by (ds string, hr 
string)
-PREHOOK: type: CREATETABLE
-PREHOOK: Output: database:default
-PREHOOK: Output: default@dummy
-POSTHOOK: query: -- partitioned table analyze 
-
-create table dummy (key string, value string) partitioned by (ds string, hr 
string)
-POSTHOOK: type: CREATETABLE
-POSTHOOK: Output: database:default
-POSTHOOK: Output: default@dummy
-PREHOOK: query: load data local inpath '../../data/files/kv1.txt' into table 
dummy partition (ds='2008',hr='12')
-PREHOOK: type: LOAD
- A masked pattern was here 
-PREHOOK: Output: default@dummy
-POSTHOOK: query: load data local inpath '../../data/files/kv1.txt' into table 
dummy partition (ds='2008',hr='12')
-POSTHOOK: type: LOAD
- A masked pattern was here 
-POSTHOOK: Output: default@dummy
-POSTHOOK: Output: default@dummy@ds=2008/hr=12
-PREHOOK: query: load data local inpath '../../data/files/kv1.txt' into table 
dummy partition (ds='2008',hr='11')
-PREHOOK: type: LOAD
- A masked pattern was here 
-PREHOOK: Output: default@dummy
-POSTHOOK: query: load data local inpath '../../data/files/kv1.txt' into table 
dummy partition (ds='2008',hr='11')
-POSTHOOK: type: LOAD
- A masked pattern was here 
-POSTHOOK: Output: default@dummy
-POSTHOOK: Output: default@dummy@ds=2008/hr=11
-PREHOOK: query: analyze table dummy partition (ds,hr) compute statistics
-PREHOOK: type: QUERY
-PREHOOK: Input: default@dummy
-PREHOOK: Input: default@dummy@ds=2008/hr=11
-PREHOOK: Input: default@dummy@ds=2008/hr=12
-PREHOOK: Output: default@dummy
-PREHOOK: Output: default@dummy@ds=2008/hr=11
-PREHOOK: Output: default@dummy@ds=2008/hr=12
-POSTHOOK: query: analyze table dummy partition (ds,hr) compute statistics
-POSTHOOK: type: QUERY
-POSTHOOK: Input: default@dummy
-POSTHOOK: Input: default@dummy@ds=2008/hr=11
-POSTHOOK: Input: default@dummy@ds=2008/hr=12
-POSTHOOK: Output: default@dummy
-POSTHOOK: Output: default@dummy@ds=2008/hr=11
-POSTHOOK: Output: default@dummy@ds=2008/hr=12
-PREHOOK: query: describe formatted dummy partition (ds='2008', hr='11')
-PREHOOK: type: DESCTABLE
-PREHOOK: Input: default@dummy
-POSTHOOK: query: describe formatted dummy partition (ds='2008', hr='11')
-POSTHOOK: type: DESCTABLE
-POSTHOOK: Input: default@dummy
-# col_name data_type   comment 
-
-keystring  
-value  string  
-
-# Partition Information 
-# col_name data_type   comment 
-
-ds string  
-hr string  
-
-# Detailed Partition Information
-Partition Value:   [2008, 11]   
-Database:  default  
-Table: dummy
- A masked pattern was here 
-Partition Parameters:   
-   COLUMN_STATS_ACCURATE   true
-   numFiles1   
-   numRows 500 
-   rawDataSize 5312
-   totalSize   5812
- A masked pattern was here 
-
-# Storage Information   
-SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe  
 
-InputFormat:   org.apache.hadoop.mapred.TextInputFormat 
-OutputFormat:  
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat   
-Compressed:No   
-Num Buckets:   -1   
-Bucket Columns:[]   
-Sort Columns:  []   
-Storage Desc Params:
-   serialization.format1   
-PREHOOK: query: describe formatted dummy partition (ds='2008', hr='12')
-PREHOOK: type: DESCTABLE
-PREHOOK: Input: default@dummy
-POSTHOOK: query: describe formatted dummy partition (ds='2008', hr='12')
-POSTHOOK: type: DESCTABLE
-POSTHOOK: Input: default@dummy
-# col_name data_type   comment 
-
-keystring