from:"\"stakiar\""

[hive] Git Push Summary

2017-11-13 Thread stakiar

Repository: hive
Updated Tags:  refs/tags/rel/release-2.3.2 [created] 7d66cfc27

[hive] Git Push Summary

2017-11-09 Thread stakiar

Repository: hive
Updated Tags:  refs/tags/release-2.3.2 [created] a0988d39a

[2/2] hive git commit: Release Notes

2017-11-09 Thread stakiar

Release Notes


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/857a9fd8
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/857a9fd8
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/857a9fd8

Branch: refs/heads/branch-2.3
Commit: 857a9fd8ad725a53bd95c1b2d6612f9b1155f44d
Parents: d10b470
Author: Sahil Takiar 
Authored: Thu Nov 9 09:00:00 2017 -0800
Committer: Sahil Takiar 
Committed: Thu Nov 9 09:00:00 2017 -0800

--
 RELEASE_NOTES.txt | 34 --
 1 file changed, 24 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/857a9fd8/RELEASE_NOTES.txt
--
diff --git a/RELEASE_NOTES.txt b/RELEASE_NOTES.txt
index c36fd66..51f3a6b 100644
--- a/RELEASE_NOTES.txt
+++ b/RELEASE_NOTES.txt
@@ -1,8 +1,8 @@
 
-Release Notes - Hive - Version 2.3.1
+Release Notes - Hive - Version 2.3.2
 
 ** Sub-task
-* [HIVE-17514] - Use SHA-256 for cookie signer to improve security
+* [HIVE-16312] - Flaky test: TestHCatClient.testTransportFailure
 
 
 
@@ -11,10 +11,28 @@ Release Notes - Hive - Version 2.3.1
 
 
 ** Bug
-* [HIVE-17562] - ACID 1.0 + ETL strategy should treat empty compacted 
files as uncovered deltas
-* [HIVE-17664] - Refactor and add new tests
-* [HIVE-17665] - Update netty-all to latest 4.0.x.Final
-* [HIVE-17705] - HIVE-17562 is returning incorrect results
+* [HIVE-10378] - Hive Update statement set keyword work with lower case 
only and doesn't give any error if wrong column name specified in the set 
clause.
+* [HIVE-15761] - ObjectStore.getNextNotification could return an empty 
NotificationEventResponse causing TProtocolException 
+* [HIVE-16213] - ObjectStore can leak Queries when rollbackTransaction 
throws an exception
+* [HIVE-16487] - Serious Zookeeper exception is logged when a race 
condition happens
+* [HIVE-16646] - Alias in transform ... as clause shouldn't be case 
sensitive
+* [HIVE-16930] - HoS should verify the value of Kerberos principal and 
keytab file before adding them to spark-submit command parameters
+* [HIVE-16991] - HiveMetaStoreClient needs a 2-arg constructor for 
backwards compatibility
+* [HIVE-17008] - Fix boolean flag switchup in DropTableEvent
+* [HIVE-17150] - CREATE INDEX execute HMS out-of-transaction listener 
calls inside a transaction
+* [HIVE-17169] - Avoid extra call to KeyProvider::getMetadata()
+* [HIVE-17184] - Unexpected new line in beeline output when running with 
-f option
+* [HIVE-17189] - Fix backwards incompatibility in HiveMetaStoreClient
+* [HIVE-17472] - Drop-partition for multi-level partition fails, if data 
does not exist.
+* [HIVE-17640] - Comparison of date return null if time part is provided 
in string.
+* [HIVE-17831] - HiveSemanticAnalyzerHookContext does not update the 
HiveOperation after sem.analyze() is called
+* [HIVE-17891] - HIVE-13076 uses create table if not exists for the 
postgres script
+* [HIVE-17928] - branch-2.3 does not compile due to using incorrect 
storage-api version
+* [HIVE-17948] - Hive 2.3.2 Release Planning
+* [HIVE-17966] - 
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveArrayInspector - Review
+
+
+
 
 
 
@@ -23,8 +41,6 @@ Release Notes - Hive - Version 2.3.1
 
 
 
-** Improvement
-* [HIVE-17048] - Pass HiveOperation info to HiveSemanticAnalyzerHook 
through HiveSemanticAnalyzerHookContext
 
 
 
@@ -44,8 +60,6 @@ Release Notes - Hive - Version 2.3.1
 
 
 
-** Task
-* [HIVE-17862] - Update copyright date in NOTICE

[1/2] hive git commit: Preparing for 2.3.2 release

2017-11-09 Thread stakiar

Repository: hive
Updated Branches:
  refs/heads/branch-2.3 62d9b1981 -> 857a9fd8a


Preparing for 2.3.2 release


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/d10b470e
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/d10b470e
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/d10b470e

Branch: refs/heads/branch-2.3
Commit: d10b470e912e34f0b825816213962ee6311ac913
Parents: 62d9b19
Author: Sahil Takiar 
Authored: Thu Nov 9 08:47:35 2017 -0800
Committer: Sahil Takiar 
Committed: Thu Nov 9 08:47:35 2017 -0800

--
 accumulo-handler/pom.xml | 2 +-
 beeline/pom.xml  | 2 +-
 cli/pom.xml  | 2 +-
 common/pom.xml   | 2 +-
 contrib/pom.xml  | 2 +-
 druid-handler/pom.xml| 2 +-
 hbase-handler/pom.xml| 2 +-
 hcatalog/core/pom.xml| 2 +-
 hcatalog/hcatalog-pig-adapter/pom.xml| 2 +-
 hcatalog/pom.xml | 2 +-
 hcatalog/server-extensions/pom.xml   | 2 +-
 hcatalog/streaming/pom.xml   | 2 +-
 hcatalog/webhcat/java-client/pom.xml | 2 +-
 hcatalog/webhcat/svr/pom.xml | 2 +-
 hplsql/pom.xml   | 2 +-
 itests/custom-serde/pom.xml  | 2 +-
 itests/custom-udfs/pom.xml   | 2 +-
 itests/custom-udfs/udf-classloader-udf1/pom.xml  | 2 +-
 itests/custom-udfs/udf-classloader-udf2/pom.xml  | 2 +-
 itests/custom-udfs/udf-classloader-util/pom.xml  | 2 +-
 itests/custom-udfs/udf-vectorized-badexample/pom.xml | 2 +-
 itests/hcatalog-unit/pom.xml | 2 +-
 itests/hive-blobstore/pom.xml| 2 +-
 itests/hive-jmh/pom.xml  | 2 +-
 itests/hive-minikdc/pom.xml  | 2 +-
 itests/hive-unit-hadoop2/pom.xml | 2 +-
 itests/hive-unit/pom.xml | 2 +-
 itests/pom.xml   | 2 +-
 itests/qtest-accumulo/pom.xml| 2 +-
 itests/qtest-spark/pom.xml   | 2 +-
 itests/qtest/pom.xml | 2 +-
 itests/test-serde/pom.xml| 2 +-
 itests/util/pom.xml  | 2 +-
 jdbc-handler/pom.xml | 2 +-
 jdbc/pom.xml | 2 +-
 llap-client/pom.xml  | 2 +-
 llap-common/pom.xml  | 2 +-
 llap-ext-client/pom.xml  | 2 +-
 llap-server/pom.xml  | 2 +-
 llap-tez/pom.xml | 2 +-
 metastore/pom.xml| 2 +-
 packaging/pom.xml| 2 +-
 pom.xml  | 2 +-
 ql/pom.xml   | 2 +-
 serde/pom.xml| 2 +-
 service-rpc/pom.xml  | 2 +-
 service/pom.xml  | 2 +-
 shims/0.23/pom.xml   | 2 +-
 shims/aggregator/pom.xml | 2 +-
 shims/common/pom.xml | 2 +-
 shims/pom.xml| 2 +-
 shims/scheduler/pom.xml  | 2 +-
 spark-client/pom.xml | 4 ++--
 testutils/pom.xml| 2 +-
 vector-code-gen/pom.xml  | 2 +-
 55 files changed, 56 insertions(+), 56 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/d10b470e/accumulo-handler/pom.xml
--
diff --git a/accumulo-handler/pom.xml b/accumulo-handler/pom.xml
index 46d5064..143a901 100644
--- a/accumulo-handler/pom.xml
+++ b/accumulo-handler/pom.xml
@@ -19,7 +19,7 @@
   
 org.apache.hive
 hive
-2.3.2-SNAPSHOT
+2.3.2
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/hive/blob/d10b470e/beeline/pom.xml
--
diff --git a/beeline/pom.xml b/beeline/pom.xml
index 4a4be28..12337b2 100644
--- a/beeline/pom.xml
+++ b/beeline/pom.xml
@@ -19,7 +19,7 @@
   
 org.apache.hive
 hive
-2.3.2-SNAPSHOT
+2.3.2
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/hive/blob/d10b470e/cl

[07/14] hive git commit: HIVE-17150: CREATE INDEX execute HMS out-of-transaction listener calls inside a transaction (Sergio Pena, reviewed by Vihang Karajgaonkar)

2017-11-08 Thread stakiar

HIVE-17150: CREATE INDEX execute HMS out-of-transaction listener calls inside a 
transaction (Sergio Pena, reviewed by Vihang Karajgaonkar)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/cd39cf38
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/cd39cf38
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/cd39cf38

Branch: refs/heads/branch-2.3
Commit: cd39cf38aae32ac39cb1adf92966be32ab796a6a
Parents: aef5ebb
Author: Sergio Pena 
Authored: Mon Jul 24 16:52:49 2017 -0500
Committer: Sahil Takiar 
Committed: Tue Nov 7 08:15:47 2017 -0800

--
 .../listener/DbNotificationListener.java|  1 +
 .../MetaStoreEventListenerConstants.java| 33 
 .../listener/DummyRawStoreFailEvent.java|  5 +++
 .../listener/TestDbNotificationListener.java|  1 +
 .../hadoop/hive/metastore/HiveMetaStore.java| 40 ++-
 .../MetaStoreEventListenerConstants.java| 41 
 .../metastore/MetaStoreListenerNotifier.java| 15 ++-
 .../apache/hadoop/hive/metastore/RawStore.java  |  2 +
 .../hadoop/hive/metastore/hbase/HBaseStore.java |  5 +++
 .../DummyRawStoreControlledCommit.java  |  5 +++
 .../DummyRawStoreForJdoConnection.java  |  5 +++
 11 files changed, 100 insertions(+), 53 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/cd39cf38/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
--
diff --git 
a/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
 
b/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
index bbfbc36..f08b970 100644
--- 
a/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
+++ 
b/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
@@ -30,6 +30,7 @@ import org.apache.hadoop.fs.PathFilter;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler;
 import org.apache.hadoop.hive.metastore.MetaStoreEventListener;
+import org.apache.hadoop.hive.metastore.MetaStoreEventListenerConstants;
 import org.apache.hadoop.hive.metastore.RawStore;
 import org.apache.hadoop.hive.metastore.RawStoreProxy;
 import org.apache.hadoop.hive.metastore.ReplChangeManager;

http://git-wip-us.apache.org/repos/asf/hive/blob/cd39cf38/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/MetaStoreEventListenerConstants.java
--
diff --git 
a/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/MetaStoreEventListenerConstants.java
 
b/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/MetaStoreEventListenerConstants.java
deleted file mode 100644
index a4f2d59..000
--- 
a/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/MetaStoreEventListenerConstants.java
+++ /dev/null
@@ -1,33 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.hive.hcatalog.listener;
-
-/**
- * Keeps a list of reserved keys used by Hive listeners when updating the 
ListenerEvent
- * parameters.
- */
-public class MetaStoreEventListenerConstants {
-  /*
-   * DbNotificationListener keys reserved for updating ListenerEvent 
parameters.
-   *
-   * DB_NOTIFICATION_EVENT_ID_KEY_NAME This key will have the event identifier 
that DbNotificationListener
-   *   processed during an event. This event 
identifier might be shared
-   *   across other MetaStoreEventListener 
implementations.
-   */
-  public static final String DB_NOTIFICATION_EVENT_ID_KEY_NAME = 
"DB_NOTIFICATION_EVENT_ID_KEY_NAME";
-}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/hive/blob/cd

[02/14] hive git commit: HIVE-17640: Comparison of date return null if time part is provided in string. (Yongzhi Chen, reviewed by Aihua Xu)

2017-11-08 Thread stakiar

HIVE-17640: Comparison of date return null if time part is provided in string. 
(Yongzhi Chen, reviewed by Aihua Xu)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/a2df8f48
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/a2df8f48
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/a2df8f48

Branch: refs/heads/branch-2.3
Commit: a2df8f48ee32008032102759d48b0172b949b494
Parents: f3dc6a9
Author: Yongzhi Chen 
Authored: Fri Oct 20 10:29:49 2017 -0400
Committer: Sahil Takiar 
Committed: Tue Nov 7 08:15:47 2017 -0800

--
 .../queries/clientpositive/date_withtimestamp.q |  3 +++
 .../clientpositive/annotate_stats_select.q.out  |  4 +--
 .../clientpositive/constantfolding.q.out|  4 +--
 .../clientpositive/date_withtimestamp.q.out | 27 
 .../PrimitiveObjectInspectorUtils.java  | 16 +---
 5 files changed, 47 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/a2df8f48/ql/src/test/queries/clientpositive/date_withtimestamp.q
--
diff --git a/ql/src/test/queries/clientpositive/date_withtimestamp.q 
b/ql/src/test/queries/clientpositive/date_withtimestamp.q
new file mode 100644
index 000..b6d04f5
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/date_withtimestamp.q
@@ -0,0 +1,3 @@
+select "2016-12-29 23:59:59"  < cast("2016-12-30" as date);
+select "2016-12-30 00:00:00"  = cast("2016-12-30" as date);
+select "2016-12-31 00:00:01"  > cast("2016-12-30" as date);

http://git-wip-us.apache.org/repos/asf/hive/blob/a2df8f48/ql/src/test/results/clientpositive/annotate_stats_select.q.out
--
diff --git a/ql/src/test/results/clientpositive/annotate_stats_select.q.out 
b/ql/src/test/results/clientpositive/annotate_stats_select.q.out
index 873f1ab..489f4d2 100644
--- a/ql/src/test/results/clientpositive/annotate_stats_select.q.out
+++ b/ql/src/test/results/clientpositive/annotate_stats_select.q.out
@@ -470,9 +470,9 @@ STAGE PLANS:
   alias: alltypes_orc
   Statistics: Num rows: 2 Data size: 1686 Basic stats: COMPLETE Column 
stats: COMPLETE
   Select Operator
-expressions: null (type: date)
+expressions: 1970-12-31 (type: date)
 outputColumnNames: _col0
-Statistics: Num rows: 2 Data size: 56 Basic stats: COMPLETE Column 
stats: COMPLETE
+Statistics: Num rows: 2 Data size: 112 Basic stats: COMPLETE 
Column stats: COMPLETE
 ListSink
 
 PREHOOK: query: explain select cast("58.174" as DECIMAL) from alltypes_orc

http://git-wip-us.apache.org/repos/asf/hive/blob/a2df8f48/ql/src/test/results/clientpositive/constantfolding.q.out
--
diff --git a/ql/src/test/results/clientpositive/constantfolding.q.out 
b/ql/src/test/results/clientpositive/constantfolding.q.out
index 10e185f..f9a9d24 100644
--- a/ql/src/test/results/clientpositive/constantfolding.q.out
+++ b/ql/src/test/results/clientpositive/constantfolding.q.out
@@ -205,9 +205,9 @@ STAGE PLANS:
   alias: src
   Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE 
Column stats: COMPLETE
   Select Operator
-expressions: null (type: date)
+expressions: 1970-12-31 (type: date)
 outputColumnNames: _col0
-Statistics: Num rows: 500 Data size: 56 Basic stats: COMPLETE 
Column stats: COMPLETE
+Statistics: Num rows: 500 Data size: 28000 Basic stats: COMPLETE 
Column stats: COMPLETE
 ListSink
 
 PREHOOK: query: CREATE TABLE dest1(c1 STRING) STORED AS TEXTFILE

http://git-wip-us.apache.org/repos/asf/hive/blob/a2df8f48/ql/src/test/results/clientpositive/date_withtimestamp.q.out
--
diff --git a/ql/src/test/results/clientpositive/date_withtimestamp.q.out 
b/ql/src/test/results/clientpositive/date_withtimestamp.q.out
new file mode 100644
index 000..3661888
--- /dev/null
+++ b/ql/src/test/results/clientpositive/date_withtimestamp.q.out
@@ -0,0 +1,27 @@
+PREHOOK: query: select "2016-12-29 23:59:59"  < cast("2016-12-30" as date)
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+ A masked pattern was here 
+POSTHOOK: query: select "2016-12-29 23:59:59"  < cast("2016-12-30" as date)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+ A masked pattern was here 
+true
+PREHOOK: query: select "2016-12-30 00:00:00"  = cast("2016-12-30" as date)
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+ A masked pattern was here 
+POSTHOOK: query: select "2016-12-30 00:00

[12/14] hive git commit: HIVE-16213: ObjectStore can leak Queries when rollbackTransaction throws an exception (Vihang Karajgaonkar, reviewed by Sergio Pena)

2017-11-08 Thread stakiar

HIVE-16213: ObjectStore can leak Queries when rollbackTransaction throws an 
exception (Vihang Karajgaonkar, reviewed by Sergio Pena)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/c2b5dba7
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/c2b5dba7
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/c2b5dba7

Branch: refs/heads/branch-2.3
Commit: c2b5dba78e2ad1c9884cd4a54f7113532614b605
Parents: 145ed20
Author: Vihang Karajgaonkar 
Authored: Tue May 2 10:27:51 2017 -0500
Committer: Sahil Takiar 
Committed: Tue Nov 7 08:15:48 2017 -0800

--
 .../hadoop/hive/metastore/ObjectStore.java  | 549 ---
 .../hadoop/hive/metastore/TestObjectStore.java  |  14 +
 2 files changed, 131 insertions(+), 432 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/c2b5dba7/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
--
diff --git 
a/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
b/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
index 358cf17..e6a918b 100644
--- a/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
+++ b/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
@@ -234,26 +234,22 @@ public class ObjectStore implements RawStore, 
Configurable {
   private Pattern partitionValidationPattern;
 
   /**
-   * A class to pass the Query object to the caller to let the caller release
-   * resources by calling QueryWrapper.query.closeAll() after consuming all 
the query results.
+   * A Autocloseable wrapper around Query class to pass the Query object to 
the caller and let the caller release
+   * the resources when the QueryWrapper goes out of scope
*/
-  public static class QueryWrapper {
+  public static class QueryWrapper implements AutoCloseable {
 public Query query;
 
 /**
  * Explicitly closes the query object to release the resources
  */
+@Override
 public void close() {
   if (query != null) {
 query.closeAll();
 query = null;
   }
 }
-
-@Override
-protected void finalize() {
-  this.close();
-}
   }
 
   public ObjectStore() {
@@ -700,12 +696,7 @@ public class ObjectStore implements RawStore, Configurable 
{
   pm.retrieve(mdb);
   commited = commitTransaction();
 } finally {
-  if (!commited) {
-rollbackTransaction();
-  }
-  if (query != null) {
-query.closeAll();
-  }
+  rollbackAndCleanup(commited, query);
 }
 if (mdb == null) {
   throw new NoSuchObjectException("There is no database named " + name);
@@ -824,10 +815,7 @@ public class ObjectStore implements RawStore, Configurable 
{
   }
   success = commitTransaction();
 } finally {
-  if (!success) {
-rollbackTransaction();
-  }
-  queryWrapper.close();
+  rollbackAndCleanup(success, queryWrapper);
 }
 return success;
   }
@@ -858,12 +846,7 @@ public class ObjectStore implements RawStore, Configurable 
{
   }
   commited = commitTransaction();
 } finally {
-  if (!commited) {
-rollbackTransaction();
-  }
-  if (query != null) {
-query.closeAll();
-  }
+  rollbackAndCleanup(commited, query);
 }
 return databases;
   }
@@ -883,12 +866,7 @@ public class ObjectStore implements RawStore, Configurable 
{
   databases = new ArrayList((Collection) query.execute());
   commited = commitTransaction();
 } finally {
-  if (!commited) {
-rollbackTransaction();
-  }
-  if (query != null) {
-query.closeAll();
-  }
+  rollbackAndCleanup(commited, query);
 }
 Collections.sort(databases);
 return databases;
@@ -956,12 +934,7 @@ public class ObjectStore implements RawStore, Configurable 
{
   }
   commited = commitTransaction();
 } finally {
-  if (!commited) {
-rollbackTransaction();
-  }
-  if (query != null) {
-query.closeAll();
-  }
+  rollbackAndCleanup(commited, query);
 }
 return type;
   }
@@ -985,12 +958,7 @@ public class ObjectStore implements RawStore, Configurable 
{
   success = commitTransaction();
   LOG.debug("type not found " + typeName, e);
 } finally {
-  if (!success) {
-rollbackTransaction();
-  }
-  if (query != null) {
-query.closeAll();
-  }
+  rollbackAndCleanup(success, query);
 }
 return success;
   }
@@ -1231,12 +1199,7 @@ public class ObjectStore implements RawStore, 
Configurable {
   }
   commited = commitTransaction();
 } finally {
-  if (!commited) {
-rollbackTransaction();
-  }
-  if (query != null) {
-

[03/14] hive git commit: HIVE-17472: Drop-partition for multi-level partition fails, if data does not exist. (Chris Drome, reviewed by Mithun Radhakrishnan)

2017-11-08 Thread stakiar

HIVE-17472: Drop-partition for multi-level partition fails, if data does not 
exist. (Chris Drome, reviewed by Mithun Radhakrishnan)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/f3dc6a97
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/f3dc6a97
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/f3dc6a97

Branch: refs/heads/branch-2.3
Commit: f3dc6a9774912257bb9ecdc9fe19ee32698fded0
Parents: b8bd459
Author: Mithun RK 
Authored: Wed Sep 6 17:25:04 2017 -0700
Committer: Sahil Takiar 
Committed: Tue Nov 7 08:15:47 2017 -0800

--
 .../hadoop/hive/metastore/HiveMetaStore.java|  6 +-
 .../clientpositive/drop_deleted_partitions.q| 18 +
 .../drop_deleted_partitions.q.out   | 74 
 3 files changed, 96 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/f3dc6a97/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
--
diff --git 
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
b/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
index 6f2b727..616314c 100644
--- a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
+++ b/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
@@ -3158,8 +3158,10 @@ public class HiveMetaStore extends ThriftHiveMetastore {
 
 }
 private void deleteParentRecursive(Path parent, int depth, boolean 
mustPurge) throws IOException, MetaException {
-  if (depth > 0 && parent != null && wh.isWritable(parent) && 
wh.isEmpty(parent)) {
-wh.deleteDir(parent, true, mustPurge);
+  if (depth > 0 && parent != null && wh.isWritable(parent)) {
+if (wh.isDir(parent) && wh.isEmpty(parent)) {
+  wh.deleteDir(parent, true, mustPurge);
+}
 deleteParentRecursive(parent.getParent(), depth - 1, mustPurge);
   }
 }

http://git-wip-us.apache.org/repos/asf/hive/blob/f3dc6a97/ql/src/test/queries/clientpositive/drop_deleted_partitions.q
--
diff --git a/ql/src/test/queries/clientpositive/drop_deleted_partitions.q 
b/ql/src/test/queries/clientpositive/drop_deleted_partitions.q
new file mode 100644
index 000..a758b1b
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/drop_deleted_partitions.q
@@ -0,0 +1,18 @@
+create database dmp;
+
+create table dmp.mp (a string) partitioned by (b string, c string) location 
'/tmp/dmp_mp';
+
+alter table dmp.mp add partition (b='1', c='1');
+
+show partitions dmp.mp;
+
+dfs -rm -R /tmp/dmp_mp/b=1;
+
+explain extended alter table dmp.mp drop partition (b='1');
+alter table dmp.mp drop partition (b='1');
+
+show partitions dmp.mp;
+
+drop table dmp.mp;
+
+drop database dmp;

http://git-wip-us.apache.org/repos/asf/hive/blob/f3dc6a97/ql/src/test/results/clientpositive/drop_deleted_partitions.q.out
--
diff --git a/ql/src/test/results/clientpositive/drop_deleted_partitions.q.out 
b/ql/src/test/results/clientpositive/drop_deleted_partitions.q.out
new file mode 100644
index 000..e543158
--- /dev/null
+++ b/ql/src/test/results/clientpositive/drop_deleted_partitions.q.out
@@ -0,0 +1,74 @@
+PREHOOK: query: create database dmp
+PREHOOK: type: CREATEDATABASE
+PREHOOK: Output: database:dmp
+POSTHOOK: query: create database dmp
+POSTHOOK: type: CREATEDATABASE
+POSTHOOK: Output: database:dmp
+ A masked pattern was here 
+PREHOOK: type: CREATETABLE
+ A masked pattern was here 
+PREHOOK: Output: database:dmp
+PREHOOK: Output: dmp@mp
+ A masked pattern was here 
+POSTHOOK: type: CREATETABLE
+ A masked pattern was here 
+POSTHOOK: Output: database:dmp
+POSTHOOK: Output: dmp@mp
+PREHOOK: query: alter table dmp.mp add partition (b='1', c='1')
+PREHOOK: type: ALTERTABLE_ADDPARTS
+PREHOOK: Output: dmp@mp
+POSTHOOK: query: alter table dmp.mp add partition (b='1', c='1')
+POSTHOOK: type: ALTERTABLE_ADDPARTS
+POSTHOOK: Output: dmp@mp
+POSTHOOK: Output: dmp@mp@b=1/c=1
+PREHOOK: query: show partitions dmp.mp
+PREHOOK: type: SHOWPARTITIONS
+PREHOOK: Input: dmp@mp
+POSTHOOK: query: show partitions dmp.mp
+POSTHOOK: type: SHOWPARTITIONS
+POSTHOOK: Input: dmp@mp
+b=1/c=1
+ A masked pattern was here 
+PREHOOK: query: explain extended alter table dmp.mp drop partition (b='1')
+PREHOOK: type: ALTERTABLE_DROPPARTS
+POSTHOOK: query: explain extended alter table dmp.mp drop partition (b='1')
+POSTHOOK: type: ALTERTABLE_DROPPARTS
+STAGE DEPENDENCIES:
+  Stage-0 is a root stage
+
+STAGE PLANS:
+  Stage: Stage-0
+  Drop Table Operator:
+Drop Table
+  table: dmp.mp
+
+PREHOOK: query: alter table dmp.mp drop

[14/14] hive git commit: HIVE-17008: Fix boolean flag switchup in DropTableEvent (Dan Burkert, reviewed by Mohit Sabharwal and Peter Vary)

2017-11-08 Thread stakiar

HIVE-17008: Fix boolean flag switchup in DropTableEvent (Dan Burkert, reviewed 
by Mohit Sabharwal and Peter Vary)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/62d9b198
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/62d9b198
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/62d9b198

Branch: refs/heads/branch-2.3
Commit: 62d9b19810935b19f36bdb88baa1e980dd6068c8
Parents: 3f0b696
Author: Peter Vary 
Authored: Fri Aug 11 12:00:33 2017 +0200
Committer: Sahil Takiar 
Committed: Tue Nov 7 08:16:09 2017 -0800

--
 .../src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/62d9b198/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
--
diff --git 
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
b/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
index 616314c..befd15e 100644
--- a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
+++ b/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
@@ -1700,7 +1700,7 @@ public class HiveMetaStore extends ThriftHiveMetastore {
 transactionalListenerResponses =
 MetaStoreListenerNotifier.notifyEvent(transactionalListeners,
   EventType.DROP_TABLE,
-  new DropTableEvent(tbl, 
deleteData, true, this),
+  new DropTableEvent(tbl, 
true, deleteData, this),
   envContext);
   }
   success = ms.commitTransaction();
@@ -1720,7 +1720,7 @@ public class HiveMetaStore extends ThriftHiveMetastore {
 if (!listeners.isEmpty()) {
   MetaStoreListenerNotifier.notifyEvent(listeners,
 EventType.DROP_TABLE,
-new DropTableEvent(tbl, 
deleteData, success, this),
+new DropTableEvent(tbl, 
success, deleteData, this),
 envContext,
 
transactionalListenerResponses, ms);
 }

[06/14] hive git commit: HIVE-16991: HiveMetaStoreClient needs a 2-arg constructor for backwards compatibility (Andrew Sherman, reviewed by Sergio Pena)

2017-11-08 Thread stakiar

HIVE-16991: HiveMetaStoreClient needs a 2-arg constructor for backwards 
compatibility (Andrew Sherman, reviewed by Sergio Pena)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/aef5ebb6
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/aef5ebb6
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/aef5ebb6

Branch: refs/heads/branch-2.3
Commit: aef5ebb678d37751546afbf6bca32d67e2f27dad
Parents: aaa440f
Author: Andrew Sherman 
Authored: Fri Jun 30 14:19:41 2017 -0500
Committer: Sahil Takiar 
Committed: Tue Nov 7 08:15:47 2017 -0800

--
 .../org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java| 4 
 1 file changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/aef5ebb6/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
--
diff --git 
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
b/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
index 7002620..c5db77f 100644
--- 
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
+++ 
b/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
@@ -128,6 +128,10 @@ public class HiveMetaStoreClient implements 
IMetaStoreClient {
 this(conf, null, true);
   }
 
+  public HiveMetaStoreClient(HiveConf conf, HiveMetaHookLoader hookLoader) 
throws MetaException {
+this(conf, hookLoader, true);
+  }
+
   public HiveMetaStoreClient(HiveConf conf, HiveMetaHookLoader hookLoader, 
Boolean allowEmbedded)
 throws MetaException {

[05/14] hive git commit: HIVE-16930: HoS should verify the value of Kerberos principal and keytab file before adding them to spark-submit command parameters (Yibing Shi via Chaoyu Tang)

2017-11-08 Thread stakiar

HIVE-16930: HoS should verify the value of Kerberos principal and keytab file 
before adding them to spark-submit command parameters (Yibing Shi via Chaoyu 
Tang)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/aaa440f3
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/aaa440f3
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/aaa440f3

Branch: refs/heads/branch-2.3
Commit: aaa440f35fb47648fa7b412280f01e939a83cb61
Parents: cf64bb9
Author: Chaoyu Tang 
Authored: Thu Jun 22 16:53:55 2017 -0400
Committer: Sahil Takiar 
Committed: Tue Nov 7 08:15:47 2017 -0800

--
 .../hive/spark/client/SparkClientImpl.java  | 35 +++-
 1 file changed, 19 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/aaa440f3/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java
--
diff --git 
a/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java 
b/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java
index d4b63f0..5c42bcc 100644
--- 
a/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java
+++ 
b/spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java
@@ -54,6 +54,7 @@ import java.util.concurrent.Future;
 import java.util.concurrent.TimeoutException;
 import java.util.concurrent.atomic.AtomicInteger;
 
+import org.apache.commons.lang3.StringUtils;
 import org.apache.hadoop.hive.conf.Constants;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
@@ -405,22 +406,24 @@ class SparkClientImpl implements SparkClient {
 String principal = 
SecurityUtil.getServerPrincipal(hiveConf.getVar(ConfVars.HIVE_SERVER2_KERBEROS_PRINCIPAL),
 "0.0.0.0");
 String keyTabFile = 
hiveConf.getVar(ConfVars.HIVE_SERVER2_KERBEROS_KEYTAB);
-if (hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SERVER2_ENABLE_DOAS)) {
-  List kinitArgv = Lists.newLinkedList();
-  kinitArgv.add("kinit");
-  kinitArgv.add(principal);
-  kinitArgv.add("-k");
-  kinitArgv.add("-t");
-  kinitArgv.add(keyTabFile + ";");
-  kinitArgv.addAll(argv);
-  argv = kinitArgv;
-} else {
-  // if doAs is not enabled, we pass the principal/keypad to 
spark-submit in order to
-  // support the possible delegation token renewal in Spark
-  argv.add("--principal");
-  argv.add(principal);
-  argv.add("--keytab");
-  argv.add(keyTabFile);
+if (StringUtils.isNotBlank(principal) && 
StringUtils.isNotBlank(keyTabFile)) {
+  if (hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SERVER2_ENABLE_DOAS)) 
{
+List kinitArgv = Lists.newLinkedList();
+kinitArgv.add("kinit");
+kinitArgv.add(principal);
+kinitArgv.add("-k");
+kinitArgv.add("-t");
+kinitArgv.add(keyTabFile + ";");
+kinitArgv.addAll(argv);
+argv = kinitArgv;
+  } else {
+// if doAs is not enabled, we pass the principal/keypad to 
spark-submit in order to
+// support the possible delegation token renewal in Spark
+argv.add("--principal");
+argv.add(principal);
+argv.add("--keytab");
+argv.add(keyTabFile);
+  }
 }
   }
   if (hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_SERVER2_ENABLE_DOAS)) {

[10/14] hive git commit: HIVE-17169: Avoid extra call to KeyProvider::getMetadata() (Mithun Radhakrishnan, reviewed by Owen O'Malley)

2017-11-08 Thread stakiar

HIVE-17169: Avoid extra call to KeyProvider::getMetadata() (Mithun 
Radhakrishnan, reviewed by Owen O'Malley)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/24dcccf3
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/24dcccf3
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/24dcccf3

Branch: refs/heads/branch-2.3
Commit: 24dcccf3030c5a6713dd901822dba9672f27de14
Parents: cd39cf3
Author: Mithun RK 
Authored: Tue Aug 15 13:52:11 2017 -0700
Committer: Sahil Takiar 
Committed: Tue Nov 7 08:15:47 2017 -0800

--
 .../apache/hadoop/hive/shims/Hadoop23Shims.java | 39 
 1 file changed, 24 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/24dcccf3/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
--
diff --git 
a/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
b/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
index 0483e91..985a5bd 100644
--- a/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
+++ b/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
@@ -39,6 +39,7 @@ import java.util.TreeMap;
 import javax.security.auth.Subject;
 import org.apache.commons.lang.StringUtils;
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.crypto.CipherSuite;
 import org.apache.hadoop.crypto.key.KeyProvider;
 import org.apache.hadoop.crypto.key.KeyProvider.Options;
 import org.apache.hadoop.crypto.key.KeyProviderCryptoExtension;
@@ -1200,6 +1201,14 @@ public class Hadoop23Shims extends HadoopShimsSecure {
   
((HdfsEncryptionShim)encryptionShim2).hdfsAdmin.getEncryptionZoneForPath(path2));
 }
 
+/**
+ * Compares two encryption key strengths.
+ *
+ * @param path1 First  path to compare
+ * @param path2 Second path to compare
+ * @return 1 if path1 is stronger; 0 if paths are equals; -1 if path1 is 
weaker.
+ * @throws IOException If an error occurred attempting to get key metadata
+ */
 @Override
 public int comparePathKeyStrength(Path path1, Path path2) throws 
IOException {
   EncryptionZone zone1, zone2;
@@ -1215,7 +1224,7 @@ public class Hadoop23Shims extends HadoopShimsSecure {
 return 1;
   }
 
-  return compareKeyStrength(zone1.getKeyName(), zone2.getKeyName());
+  return compareKeyStrength(zone1, zone2);
 }
 
 @Override
@@ -1267,28 +1276,28 @@ public class Hadoop23Shims extends HadoopShimsSecure {
 /**
  * Compares two encryption key strengths.
  *
- * @param keyname1 Keyname to compare
- * @param keyname2 Keyname to compare
- * @return 1 if path1 is stronger; 0 if paths are equals; -1 if path1 is 
weaker.
+ * @param zone1 First  EncryptionZone to compare
+ * @param zone2 Second EncryptionZone to compare
+ * @return 1 if zone1 is stronger; 0 if zones are equal; -1 if zone1 is 
weaker.
  * @throws IOException If an error occurred attempting to get key metadata
  */
-private int compareKeyStrength(String keyname1, String keyname2) throws 
IOException {
-  KeyProvider.Metadata meta1, meta2;
+private int compareKeyStrength(EncryptionZone zone1, EncryptionZone zone2) 
throws IOException {
 
-  if (keyProvider == null) {
-throw new IOException("HDFS security key provider is not configured on 
your server.");
-  }
+  // zone1, zone2 should already have been checked for nulls.
+  assert zone1 != null && zone2 != null : "Neither EncryptionZone under 
comparison can be null.";
 
-  meta1 = keyProvider.getMetadata(keyname1);
-  meta2 = keyProvider.getMetadata(keyname2);
+  CipherSuite suite1 = zone1.getSuite();
+  CipherSuite suite2 = zone2.getSuite();
 
-  if (meta1.getBitLength() < meta2.getBitLength()) {
-return -1;
-  } else if (meta1.getBitLength() == meta2.getBitLength()) {
+  if (suite1 == null && suite2 == null) {
 return 0;
-  } else {
+  } else if (suite1 == null) {
+return -1;
+  } else if (suite2 == null) {
 return 1;
   }
+
+  return Integer.compare(suite1.getAlgorithmBlockSize(), 
suite2.getAlgorithmBlockSize());
 }
   }

[09/14] hive git commit: HIVE-17891 : HIVE-13076 uses create table if not exists for the postgres script (Vihang Karajgaonkar, reviewed by Naveen Gangam)

2017-11-08 Thread stakiar

HIVE-17891 : HIVE-13076 uses create table if not exists for the postgres script 
(Vihang Karajgaonkar, reviewed by Naveen Gangam)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/ecf97040
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/ecf97040
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/ecf97040

Branch: refs/heads/branch-2.3
Commit: ecf97040a4ff64cfcba37f7fc111cdd995ee8e0f
Parents: a2df8f4
Author: Vihang Karajgaonkar 
Authored: Tue Oct 24 12:32:37 2017 -0700
Committer: Sahil Takiar 
Committed: Tue Nov 7 08:15:47 2017 -0800

--
 metastore/scripts/upgrade/postgres/033-HIVE-13076.postgres.sql | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/ecf97040/metastore/scripts/upgrade/postgres/033-HIVE-13076.postgres.sql
--
diff --git a/metastore/scripts/upgrade/postgres/033-HIVE-13076.postgres.sql 
b/metastore/scripts/upgrade/postgres/033-HIVE-13076.postgres.sql
index 9ee7c11..59e702b 100644
--- a/metastore/scripts/upgrade/postgres/033-HIVE-13076.postgres.sql
+++ b/metastore/scripts/upgrade/postgres/033-HIVE-13076.postgres.sql
@@ -1,4 +1,4 @@
-CREATE TABLE IF NOT EXISTS  "KEY_CONSTRAINTS"
+CREATE TABLE "KEY_CONSTRAINTS"
 (
   "CHILD_CD_ID" BIGINT,
   "CHILD_INTEGER_IDX" BIGINT,

[11/14] hive git commit: HIVE-17189 : Fix backwards incompatibility in HiveMetaStoreClient (Vihang Karajgaonkar, reviewed by Alan Gates)

2017-11-08 Thread stakiar

HIVE-17189 : Fix backwards incompatibility in HiveMetaStoreClient (Vihang 
Karajgaonkar, reviewed by Alan Gates)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/b8bd4594
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/b8bd4594
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/b8bd4594

Branch: refs/heads/branch-2.3
Commit: b8bd4594bef718b1eeac9fceb437d7df7b480ed1
Parents: 0ff4d5b
Author: Vihang Karajgaonkar 
Authored: Thu Jul 27 16:23:47 2017 -0700
Committer: Sahil Takiar 
Committed: Tue Nov 7 08:15:47 2017 -0800

--
 .../hive/metastore/TestHiveMetaStore.java   | 49 ++
 .../hive/metastore/HiveMetaStoreClient.java | 23 +
 .../hadoop/hive/metastore/IMetaStoreClient.java | 52 +++-
 .../ql/metadata/SessionHiveMetaStoreClient.java | 13 +
 4 files changed, 136 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/b8bd4594/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
--
diff --git 
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
 
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
index 1b0b537..4b1df8e 100644
--- 
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
+++ 
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
@@ -608,6 +608,55 @@ public abstract class TestHiveMetaStore extends TestCase {
 assertEquals(" should have returned 50 partitions", maxParts, 
partitions.size());
   }
 
+  public void testAlterTableCascade() throws Throwable {
+// create a table with multiple partitions
+String dbName = "compdb";
+String tblName = "comptbl";
+String typeName = "Person";
+
+cleanUp(dbName, tblName, typeName);
+
+List> values = new ArrayList>();
+values.add(makeVals("2008-07-01 14:13:12", "14"));
+values.add(makeVals("2008-07-01 14:13:12", "15"));
+values.add(makeVals("2008-07-02 14:13:12", "15"));
+values.add(makeVals("2008-07-03 14:13:12", "151"));
+
+createMultiPartitionTableSchema(dbName, tblName, typeName, values);
+Table tbl = client.getTable(dbName, tblName);
+List cols = tbl.getSd().getCols();
+cols.add(new FieldSchema("new_col", serdeConstants.STRING_TYPE_NAME, ""));
+tbl.getSd().setCols(cols);
+//add new column with cascade option
+client.alter_table(dbName, tblName, tbl, true);
+//
+Table tbl2 = client.getTable(dbName, tblName);
+Assert.assertEquals("Unexpected number of cols", 3, 
tbl2.getSd().getCols().size());
+Assert.assertEquals("Unexpected column name", "new_col", 
tbl2.getSd().getCols().get(2).getName());
+//get a partition
+List pvalues = new ArrayList<>(2);
+pvalues.add("2008-07-01 14:13:12");
+pvalues.add("14");
+Partition partition = client.getPartition(dbName, tblName, pvalues);
+Assert.assertEquals("Unexpected number of cols", 3, 
partition.getSd().getCols().size());
+Assert.assertEquals("Unexpected column name", "new_col", 
partition.getSd().getCols().get(2).getName());
+
+//add another column
+cols = tbl.getSd().getCols();
+cols.add(new FieldSchema("new_col2", serdeConstants.STRING_TYPE_NAME, ""));
+tbl.getSd().setCols(cols);
+//add new column with no cascade option
+client.alter_table(dbName, tblName, tbl, false);
+tbl2 = client.getTable(dbName, tblName);
+Assert.assertEquals("Unexpected number of cols", 4, 
tbl2.getSd().getCols().size());
+Assert.assertEquals("Unexpected column name", "new_col2", 
tbl2.getSd().getCols().get(3).getName());
+//get partition, this partition should not have the newly added column 
since cascade option
+//was false
+partition = client.getPartition(dbName, tblName, pvalues);
+Assert.assertEquals("Unexpected number of cols", 3, 
partition.getSd().getCols().size());  
+  }
+
+
   public void testListPartitionNames() throws Throwable {
 // create a table with multiple partitions
 String dbName = "compdb";

http://git-wip-us.apache.org/repos/asf/hive/blob/b8bd4594/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
--
diff --git 
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
b/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
index c5db77f..b1a9782 100644
--- 
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
+++ 
b/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
@@ -50,6 +50,7 @@ import java.security.PrivilegedExceptionAction;

[04/14] hive git commit: HIVE-16646: Alias in transform ... as clause shouldn't be case sensitive (Yibing Shi, reviewed by Yongzhi Chen)

2017-11-08 Thread stakiar

HIVE-16646: Alias in transform ... as clause shouldn't be case sensitive 
(Yibing Shi, reviewed by Yongzhi Chen)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/cf64bb9d
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/cf64bb9d
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/cf64bb9d

Branch: refs/heads/branch-2.3
Commit: cf64bb9d4b280b433db82a93ceace1a6091a0506
Parents: a4e3e35
Author: Yongzhi Chen 
Authored: Thu May 11 14:53:57 2017 -0400
Committer: Sahil Takiar 
Committed: Tue Nov 7 08:15:47 2017 -0800

--
 .../hadoop/hive/ql/parse/SemanticAnalyzer.java  |  4 +--
 ql/src/test/queries/clientpositive/transform3.q |  6 +
 .../results/clientpositive/transform3.q.out | 28 
 3 files changed, 36 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/cf64bb9d/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
--
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
b/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
index d49708c..2a62062 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
@@ -3770,7 +3770,7 @@ public class SemanticAnalyzer extends 
BaseSemanticAnalyzer {
   if (outputColNames) {
 for (int i = 0; i < ccount; ++i) {
   String colAlias = unescapeIdentifier(((ASTNode) collist.getChild(i))
-  .getText());
+  .getText()).toLowerCase();
   failIfColAliasExists(colAliasNamesDuplicateCheck, colAlias);
   String intName = getColumnInternalName(i);
   ColumnInfo colInfo = new ColumnInfo(intName,
@@ -3783,7 +3783,7 @@ public class SemanticAnalyzer extends 
BaseSemanticAnalyzer {
   ASTNode child = (ASTNode) collist.getChild(i);
   assert child.getType() == HiveParser.TOK_TABCOL;
   String colAlias = unescapeIdentifier(((ASTNode) child.getChild(0))
-  .getText());
+  .getText()).toLowerCase();
   failIfColAliasExists(colAliasNamesDuplicateCheck, colAlias);
   String intName = getColumnInternalName(i);
   ColumnInfo colInfo = new ColumnInfo(intName, TypeInfoUtils

http://git-wip-us.apache.org/repos/asf/hive/blob/cf64bb9d/ql/src/test/queries/clientpositive/transform3.q
--
diff --git a/ql/src/test/queries/clientpositive/transform3.q 
b/ql/src/test/queries/clientpositive/transform3.q
new file mode 100644
index 000..4a2a368
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/transform3.q
@@ -0,0 +1,6 @@
+CREATE TABLE transform3_t1 (col string);
+INSERT OVERWRITE TABLE transform3_t1 VALUES('');
+
+SELECT t.newCol FROM (
+  SELECT TRANSFORM(col) USING 'cat' AS (NewCol string) FROM transform3_t1
+) t;

http://git-wip-us.apache.org/repos/asf/hive/blob/cf64bb9d/ql/src/test/results/clientpositive/transform3.q.out
--
diff --git a/ql/src/test/results/clientpositive/transform3.q.out 
b/ql/src/test/results/clientpositive/transform3.q.out
new file mode 100644
index 000..5f93ed8
--- /dev/null
+++ b/ql/src/test/results/clientpositive/transform3.q.out
@@ -0,0 +1,28 @@
+PREHOOK: query: CREATE TABLE transform3_t1 (col string)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@transform3_t1
+POSTHOOK: query: CREATE TABLE transform3_t1 (col string)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@transform3_t1
+PREHOOK: query: INSERT OVERWRITE TABLE transform3_t1 VALUES('')
+PREHOOK: type: QUERY
+PREHOOK: Output: default@transform3_t1
+POSTHOOK: query: INSERT OVERWRITE TABLE transform3_t1 VALUES('')
+POSTHOOK: type: QUERY
+POSTHOOK: Output: default@transform3_t1
+POSTHOOK: Lineage: transform3_t1.col SIMPLE 
[(values__tmp__table__1)values__tmp__table__1.FieldSchema(name:tmp_values_col1, 
type:string, comment:), ]
+PREHOOK: query: SELECT t.newCol FROM (
+  SELECT TRANSFORM(col) USING 'cat' AS (NewCol string) FROM transform3_t1
+) t
+PREHOOK: type: QUERY
+PREHOOK: Input: default@transform3_t1
+ A masked pattern was here 
+POSTHOOK: query: SELECT t.newCol FROM (
+  SELECT TRANSFORM(col) USING 'cat' AS (NewCol string) FROM transform3_t1
+) t
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@transform3_t1
+ A masked pattern was here 
+

[13/14] hive git commit: HIVE-16487: Serious Zookeeper exception is logged when a race condition happens (Peter Vary via Chaoyu Tang)

2017-11-08 Thread stakiar

HIVE-16487: Serious Zookeeper exception is logged when a race condition happens 
(Peter Vary via Chaoyu Tang)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/3f0b696f
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/3f0b696f
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/3f0b696f

Branch: refs/heads/branch-2.3
Commit: 3f0b696f1badc10453dd59a65ce70d41420d02d1
Parents: c2b5dba
Author: Chaoyu Tang 
Authored: Mon May 1 09:53:14 2017 -0400
Committer: Sahil Takiar 
Committed: Tue Nov 7 08:15:48 2017 -0800

--
 .../zookeeper/ZooKeeperHiveLockManager.java | 39 +---
 1 file changed, 26 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/3f0b696f/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
--
diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
 
b/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
index c2a4806..9b46ae7 100644
--- 
a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
+++ 
b/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
@@ -285,8 +285,10 @@ public class ZooKeeperHiveLockManager implements 
HiveLockManager {
 int tryNum = 0;
 ZooKeeperHiveLock ret = null;
 Set conflictingLocks = new HashSet();
+Exception lastException = null;
 
 do {
+  lastException = null;
   tryNum++;
   try {
 if (tryNum > 1) {
@@ -298,26 +300,22 @@ public class ZooKeeperHiveLockManager implements 
HiveLockManager {
   break;
 }
   } catch (Exception e1) {
+lastException = e1;
 if (e1 instanceof KeeperException) {
   KeeperException e = (KeeperException) e1;
   switch (e.code()) {
   case CONNECTIONLOSS:
   case OPERATIONTIMEOUT:
+  case NONODE:
+  case NODEEXISTS:
 LOG.debug("Possibly transient ZooKeeper exception: ", e);
-continue;
+break;
   default:
 LOG.error("Serious Zookeeper exception: ", e);
 break;
   }
-}
-if (tryNum >= numRetriesForLock) {
-  console.printError("Unable to acquire " + key.getData().getLockMode()
-  + ", " + mode + " lock " + key.getDisplayName() + " after "
-  + tryNum + " attempts.");
-  LOG.error("Exceeds maximum retries with errors: ", e1);
-  printConflictingLocks(key,mode,conflictingLocks);
-  conflictingLocks.clear();
-  throw new LockException(e1);
+} else {
+  LOG.error("Other unexpected exception: ", e1);
 }
   }
 } while (tryNum < numRetriesForLock);
@@ -327,8 +325,11 @@ public class ZooKeeperHiveLockManager implements 
HiveLockManager {
   + ", " + mode + " lock " + key.getDisplayName() + " after "
   + tryNum + " attempts.");
   printConflictingLocks(key,mode,conflictingLocks);
+  if (lastException != null) {
+LOG.error("Exceeds maximum retries with errors: ", lastException);
+throw new LockException(lastException);
+  }
 }
-conflictingLocks.clear();
 return ret;
   }
 
@@ -350,6 +351,19 @@ public class ZooKeeperHiveLockManager implements 
HiveLockManager {
 }
   }
 
+  /**
+   * Creates a primitive lock object on ZooKeeper.
+   * @param key The lock data
+   * @param mode The lock mode (HiveLockMode - EXCLUSIVE/SHARED/SEMI_SHARED)
+   * @param keepAlive If true creating PERSISTENT ZooKeeper locks, otherwise 
EPHEMERAL ZooKeeper
+   *  locks
+   * @param parentCreated If we expect, that the parent is already created 
then true, otherwise
+   *  we will try to create the parents as well
+   * @param conflictingLocks The set where we should collect the conflicting 
locks when
+   * the logging level is set to DEBUG
+   * @return The created ZooKeeperHiveLock object, null if there was a 
conflicting lock
+   * @throws Exception If there was an unexpected Exception
+   */
   private ZooKeeperHiveLock lockPrimitive(HiveLockObject key,
   HiveLockMode mode, boolean keepAlive, boolean parentCreated,
   Set conflictingLocks)
@@ -390,7 +404,7 @@ public class ZooKeeperHiveLockManager implements 
HiveLockManager {
 int seqNo = getSequenceNumber(res, getLockName(lastName, mode));
 if (seqNo == -1) {
   curatorFramework.delete().forPath(res);
-  return null;
+  throw new LockException("The created node does not contain a sequence 
number: " + res);
 }
 
 List children = curatorFramework.getChildren().forPath(lastN

[01/14] hive git commit: HIVE-15761: ObjectStore.getNextNotification could return an empty NotificationEventResponse causing TProtocolException (Sergio Pena, reviewed by Aihua Xu)

2017-11-08 Thread stakiar

Repository: hive
Updated Branches:
  refs/heads/branch-2.3 a4e3e3592 -> 62d9b1981


HIVE-15761: ObjectStore.getNextNotification could return an empty 
NotificationEventResponse causing TProtocolException (Sergio Pena, reviewed by 
Aihua Xu)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/145ed20b
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/145ed20b
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/145ed20b

Branch: refs/heads/branch-2.3
Commit: 145ed20b04e4bf6bcb833e94efe71bcbe0851500
Parents: ecf9704
Author: Sergio Pena 
Authored: Fri Apr 21 15:23:52 2017 -0500
Committer: Sahil Takiar 
Committed: Tue Nov 7 08:15:47 2017 -0800

--
 .../java/org/apache/hadoop/hive/metastore/ObjectStore.java| 7 ---
 .../org/apache/hadoop/hive/metastore/TestObjectStore.java | 5 +
 2 files changed, 9 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/145ed20b/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
--
diff --git 
a/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
b/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
index a63519a..358cf17 100644
--- a/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
+++ b/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
@@ -8260,6 +8260,9 @@ public class ObjectStore implements RawStore, 
Configurable {
   public NotificationEventResponse 
getNextNotification(NotificationEventRequest rqst) {
 boolean commited = false;
 Query query = null;
+
+NotificationEventResponse result = new NotificationEventResponse();
+result.setEvents(new ArrayList());
 try {
   openTransaction();
   long lastEvent = rqst.getLastEvent();
@@ -8269,11 +8272,9 @@ public class ObjectStore implements RawStore, 
Configurable {
   Collection events = (Collection) 
query.execute(lastEvent);
   commited = commitTransaction();
   if (events == null) {
-return null;
+return result;
   }
   Iterator i = events.iterator();
-  NotificationEventResponse result = new NotificationEventResponse();
-  result.setEvents(new ArrayList());
   int maxEvents = rqst.getMaxEvents() > 0 ? rqst.getMaxEvents() : 
Integer.MAX_VALUE;
   int numEvents = 0;
   while (i.hasNext() && numEvents++ < maxEvents) {

http://git-wip-us.apache.org/repos/asf/hive/blob/145ed20b/metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java
--
diff --git 
a/metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java 
b/metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java
index 9b8eaf2..6524ee7 100644
--- a/metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java
+++ b/metastore/src/test/org/apache/hadoop/hive/metastore/TestObjectStore.java
@@ -153,11 +153,16 @@ public class TestObjectStore {
 Assert.assertEquals(2, eventResponse.getEventsSize());
 Assert.assertEquals(FIRST_EVENT_ID, 
eventResponse.getEvents().get(0).getEventId());
 Assert.assertEquals(SECOND_EVENT_ID, 
eventResponse.getEvents().get(1).getEventId());
+
 // Verify that getNextNotification(last) returns events after a specified 
event
 eventResponse = objectStore.getNextNotification(new 
NotificationEventRequest(FIRST_EVENT_ID));
 Assert.assertEquals(1, eventResponse.getEventsSize());
 Assert.assertEquals(SECOND_EVENT_ID, 
eventResponse.getEvents().get(0).getEventId());
 
+// Verify that getNextNotification(last) returns zero events if there are 
no more notifications available
+eventResponse = objectStore.getNextNotification(new 
NotificationEventRequest(SECOND_EVENT_ID));
+Assert.assertEquals(0, eventResponse.getEventsSize());
+
 // Verify that cleanNotificationEvents() cleans up all old notifications
 Thread.sleep(1);
 objectStore.cleanNotificationEvents(1);

[08/14] hive git commit: HIVE-17184 : Unexpected new line in beeline when running with -f option (Vihang Karajgaonkar, reviewed by Peter Vary)

2017-11-08 Thread stakiar

HIVE-17184 : Unexpected new line in beeline when running with -f option (Vihang 
Karajgaonkar, reviewed by Peter Vary)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/0ff4d5b8
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/0ff4d5b8
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/0ff4d5b8

Branch: refs/heads/branch-2.3
Commit: 0ff4d5b8e28d142a3fe7b58126eb4c1a6a4d4bde
Parents: 24dcccf
Author: Vihang Karajgaonkar 
Authored: Thu Jul 27 10:38:35 2017 -0700
Committer: Sahil Takiar 
Committed: Tue Nov 7 08:15:47 2017 -0800

--
 beeline/src/java/org/apache/hive/beeline/BeeLine.java | 1 -
 1 file changed, 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/0ff4d5b8/beeline/src/java/org/apache/hive/beeline/BeeLine.java
--
diff --git a/beeline/src/java/org/apache/hive/beeline/BeeLine.java 
b/beeline/src/java/org/apache/hive/beeline/BeeLine.java
index 11526a7..0696345 100644
--- a/beeline/src/java/org/apache/hive/beeline/BeeLine.java
+++ b/beeline/src/java/org/apache/hive/beeline/BeeLine.java
@@ -1158,7 +1158,6 @@ public class BeeLine implements Closeable {
   return ERRNO_OTHER;
 } finally {
   IOUtils.closeStream(fileStream);
-  output("");   // dummy new line
 }
   }

hive git commit: HIVE-17008: Fix boolean flag switchup in DropTableEvent (Dan Burkert, reviewed by Mohit Sabharwal and Peter Vary)

2017-11-06 Thread stakiar

Repository: hive
Updated Branches:
  refs/heads/branch-2 307f58270 -> d9ed95677


HIVE-17008: Fix boolean flag switchup in DropTableEvent (Dan Burkert, reviewed 
by Mohit Sabharwal and Peter Vary)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/d9ed9567
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/d9ed9567
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/d9ed9567

Branch: refs/heads/branch-2
Commit: d9ed9567720537e33ea823e851adbab93833e311
Parents: 307f582
Author: Peter Vary 
Authored: Fri Aug 11 12:00:33 2017 +0200
Committer: Sahil Takiar 
Committed: Mon Nov 6 20:25:33 2017 -0800

--
 .../src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/d9ed9567/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
--
diff --git 
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
b/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
index d52f049..d32e89c 100644
--- a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
+++ b/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
@@ -1712,7 +1712,7 @@ public class HiveMetaStore extends ThriftHiveMetastore {
 transactionalListenerResponses =
 MetaStoreListenerNotifier.notifyEvent(transactionalListeners,
   EventType.DROP_TABLE,
-  new DropTableEvent(tbl, 
deleteData, true, this),
+  new DropTableEvent(tbl, 
true, deleteData, this),
   envContext);
   }
   success = ms.commitTransaction();
@@ -1732,7 +1732,7 @@ public class HiveMetaStore extends ThriftHiveMetastore {
 if (!listeners.isEmpty()) {
   MetaStoreListenerNotifier.notifyEvent(listeners,
 EventType.DROP_TABLE,
-new DropTableEvent(tbl, 
deleteData, success, this),
+new DropTableEvent(tbl, 
success, deleteData, this),
 envContext,
 
transactionalListenerResponses, ms);
 }

hive git commit: HIVE-17953: Metrics should move to destination atomically (Alexander Kolbasov, reviewed by Sahil Takiar, Barna Zsombor Klara)

2017-11-06 Thread stakiar

Repository: hive
Updated Branches:
  refs/heads/master ddce801f2 -> d7d96658c


HIVE-17953: Metrics should move to destination atomically (Alexander Kolbasov, 
reviewed by Sahil Takiar, Barna Zsombor Klara)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/d7d96658
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/d7d96658
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/d7d96658

Branch: refs/heads/master
Commit: d7d96658c233b100754884985860bf40f97264ee
Parents: ddce801
Author: Alexander Kolbasov 
Authored: Mon Nov 6 12:49:47 2017 -0800
Committer: Sahil Takiar 
Committed: Mon Nov 6 12:50:46 2017 -0800

--
 .../hive/common/metrics/metrics2/JsonFileMetricsReporter.java   | 4 ++--
 .../org/apache/hadoop/hive/metastore/metrics/JsonReporter.java  | 5 +++--
 2 files changed, 5 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/d7d96658/common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
--
diff --git 
a/common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
 
b/common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
index b9be8bd..2e6420c 100644
--- 
a/common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
+++ 
b/common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
@@ -170,9 +170,9 @@ public class JsonFileMetricsReporter implements 
CodahaleReporter, Runnable {
 return;
   }
 
-  // Move temp file to the destination file
+  // Atomically move temp file to the destination file
   try {
-Files.move(tmpFile, path, StandardCopyOption.REPLACE_EXISTING);
+Files.move(tmpFile, path, StandardCopyOption.ATOMIC_MOVE);
   } catch (Exception e) {
 LOGGER.error("Unable to rename temp file {} to {}", tmpFile, path);
 LOGGER.error("Exception during rename", e);

http://git-wip-us.apache.org/repos/asf/hive/blob/d7d96658/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
--
diff --git 
a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
 
b/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
index 04a5f02..e198fbc 100644
--- 
a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
+++ 
b/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
@@ -163,9 +163,10 @@ public class JsonReporter extends ScheduledReporter {
 return;
   }
 
+  // Atomically move temp file to the destination file
   try {
-Files.move(tmpFile, path, StandardCopyOption.REPLACE_EXISTING);
-  } catch (IOException e) {
+Files.move(tmpFile, path, StandardCopyOption.ATOMIC_MOVE);
+  } catch (Exception e) {
 LOG.error("Unable to rename temp file {} to {}", tmpFile, path);
 LOG.error("Exception during rename", e);
   }

hive git commit: HIVE-17635: Add unit tests to CompactionTxnHandler and use PreparedStatements for queries (Andrew Sherman, reviewed by Sahil Takiar)

2017-10-31 Thread stakiar

Repository: hive
Updated Branches:
  refs/heads/master 5b8ffe2d9 -> 2a2f64270


HIVE-17635: Add unit tests to CompactionTxnHandler and use PreparedStatements 
for queries (Andrew Sherman, reviewed by Sahil Takiar)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/2a2f6427
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/2a2f6427
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/2a2f6427

Branch: refs/heads/master
Commit: 2a2f6427014045b9119714d205d7b8face9f7d92
Parents: 5b8ffe2
Author: Andrew Sherman 
Authored: Tue Oct 31 18:57:52 2017 -0700
Committer: Sahil Takiar 
Committed: Tue Oct 31 18:57:52 2017 -0700

--
 .../org/apache/hive/beeline/HiveSchemaTool.java |  34 ++--
 .../metastore/txn/TestCompactionTxnHandler.java |  63 +++
 .../metastore/txn/CompactionTxnHandler.java | 168 +--
 .../hadoop/hive/metastore/txn/TxnUtils.java |  52 +-
 .../hadoop/hive/metastore/txn/TestTxnUtils.java |  38 -
 5 files changed, 280 insertions(+), 75 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/2a2f6427/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java
--
diff --git a/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java 
b/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java
index 5350311..04576ae 100644
--- a/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java
+++ b/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java
@@ -667,27 +667,31 @@ public class HiveSchemaTool {
   for (String seqName : seqNameToTable.keySet()) {
 String tableName = seqNameToTable.get(seqName).getLeft();
 String tableKey = seqNameToTable.get(seqName).getRight();
+String fullSequenceName = "org.apache.hadoop.hive.metastore.model." + 
seqName;
 String seqQuery = needsQuotedIdentifier ?
-("select t.\"NEXT_VAL\" from \"SEQUENCE_TABLE\" t WHERE 
t.\"SEQUENCE_NAME\"='org.apache.hadoop.hive.metastore.model." + seqName + "' 
order by t.\"SEQUENCE_NAME\" ")
-: ("select t.NEXT_VAL from SEQUENCE_TABLE t WHERE 
t.SEQUENCE_NAME='org.apache.hadoop.hive.metastore.model." + seqName + "' order 
by t.SEQUENCE_NAME ");
+("select t.\"NEXT_VAL\" from \"SEQUENCE_TABLE\" t WHERE 
t.\"SEQUENCE_NAME\"=? order by t.\"SEQUENCE_NAME\" ")
+: ("select t.NEXT_VAL from SEQUENCE_TABLE t WHERE 
t.SEQUENCE_NAME=? order by t.SEQUENCE_NAME ");
 String maxIdQuery = needsQuotedIdentifier ?
 ("select max(\"" + tableKey + "\") from \"" + tableName + "\"")
 : ("select max(" + tableKey + ") from " + tableName);
 
-  ResultSet res = stmt.executeQuery(maxIdQuery);
-  if (res.next()) {
- long maxId = res.getLong(1);
- if (maxId > 0) {
-   ResultSet resSeq = stmt.executeQuery(seqQuery);
-   if (!resSeq.next()) {
- isValid = false;
- System.err.println("Missing SEQUENCE_NAME " + seqName + " 
from SEQUENCE_TABLE");
-   } else if (resSeq.getLong(1) < maxId) {
- isValid = false;
- System.err.println("NEXT_VAL for " + seqName + " in 
SEQUENCE_TABLE < max("+ tableKey + ") in " + tableName);
-   }
- }
+ResultSet res = stmt.executeQuery(maxIdQuery);
+if (res.next()) {
+  long maxId = res.getLong(1);
+  if (maxId > 0) {
+PreparedStatement pStmt = conn.prepareStatement(seqQuery);
+pStmt.setString(1, fullSequenceName);
+ResultSet resSeq = pStmt.executeQuery();
+if (!resSeq.next()) {
+  isValid = false;
+  System.err.println("Missing SEQUENCE_NAME " + seqName + " from 
SEQUENCE_TABLE");
+} else if (resSeq.getLong(1) < maxId) {
+  isValid = false;
+  System.err.println("NEXT_VAL for " + seqName + " in 
SEQUENCE_TABLE < max(" +
+  tableKey + ") in " + tableName);
+}
   }
+}
   }
 
   System.out.println((isValid ? "Succeeded" :"Failed") + " in sequence 
number validation for SEQUENCE_TABLE.");

http://git-wip-us.apache.org/repos/asf/hive/blob/2a2f6427/ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
--
diff --git 
a/ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
 
b/ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
index 96005b4..34a1600 100644
--- 
a/ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
+++ 
b/ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java
@@

hive git commit: HIVE-17868: Make queries in spark_local_queries.q have deterministic output (Andrew Sherman, reviewed by Xuefu Zhang, Sahil Takiar)

2017-10-30 Thread stakiar

Repository: hive
Updated Branches:
  refs/heads/master d085bc274 -> ed92b154c


HIVE-17868: Make queries in spark_local_queries.q have deterministic output 
(Andrew Sherman, reviewed by Xuefu Zhang, Sahil Takiar)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/ed92b154
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/ed92b154
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/ed92b154

Branch: refs/heads/master
Commit: ed92b154c5362d405f0b4793c2e94e842dfd35c0
Parents: d085bc2
Author: Andrew Sherman 
Authored: Mon Oct 30 10:23:41 2017 -0700
Committer: Sahil Takiar 
Committed: Mon Oct 30 10:23:41 2017 -0700

--
 .../clientpositive/spark_local_queries.q|  6 +-
 .../spark/spark_local_queries.q.out | 60 ++--
 2 files changed, 33 insertions(+), 33 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/ed92b154/ql/src/test/queries/clientpositive/spark_local_queries.q
--
diff --git a/ql/src/test/queries/clientpositive/spark_local_queries.q 
b/ql/src/test/queries/clientpositive/spark_local_queries.q
index 265d922..8238113 100644
--- a/ql/src/test/queries/clientpositive/spark_local_queries.q
+++ b/ql/src/test/queries/clientpositive/spark_local_queries.q
@@ -7,10 +7,10 @@ explain select * from src order by key limit 10;
 select * from src order by key limit 10;
 
 -- Test join
-select * from src join src1 on src.key = src1.key limit 10;
+select * from src join src1 on src.key = src1.key order by src.key limit 10;
 
 -- Test filer on partitioned table
-select * from srcpart where ds = "2008-04-08" limit 10;
+select * from srcpart where ds = "2008-04-08" order by key limit 10;
 
 -- Test group by
-select key, count(*) from src group by key limit 10;
+select key, count(*) from src group by key order by key limit 10;

http://git-wip-us.apache.org/repos/asf/hive/blob/ed92b154/ql/src/test/results/clientpositive/spark/spark_local_queries.q.out
--
diff --git a/ql/src/test/results/clientpositive/spark/spark_local_queries.q.out 
b/ql/src/test/results/clientpositive/spark/spark_local_queries.q.out
index 770369c..f34c5fa 100644
--- a/ql/src/test/results/clientpositive/spark/spark_local_queries.q.out
+++ b/ql/src/test/results/clientpositive/spark/spark_local_queries.q.out
@@ -69,63 +69,63 @@ POSTHOOK: Input: default@src
 103val_103
 104val_104
 104val_104
-PREHOOK: query: select * from src join src1 on src.key = src1.key limit 10
+PREHOOK: query: select * from src join src1 on src.key = src1.key order by 
src.key limit 10
 PREHOOK: type: QUERY
 PREHOOK: Input: default@src
 PREHOOK: Input: default@src1
  A masked pattern was here 
-POSTHOOK: query: select * from src join src1 on src.key = src1.key limit 10
+POSTHOOK: query: select * from src join src1 on src.key = src1.key order by 
src.key limit 10
 POSTHOOK: type: QUERY
 POSTHOOK: Input: default@src
 POSTHOOK: Input: default@src1
  A masked pattern was here 
-213val_213 213 val_213
-213val_213 213 val_213
-150val_150 150 val_150
-238val_238 238 val_238
-238val_238 238 val_238
+128val_128 128 
+128val_128 128 
+128val_128 128 
 146val_146 146 val_146
 146val_146 146 val_146
-255val_255 255 val_255
-255val_255 255 val_255
-401val_401 401 val_401
-PREHOOK: query: select * from srcpart where ds = "2008-04-08" limit 10
+150val_150 150 val_150
+213val_213 213 val_213
+213val_213 213 val_213
+224val_224 224 
+224val_224 224 
+PREHOOK: query: select * from srcpart where ds = "2008-04-08" order by key 
limit 10
 PREHOOK: type: QUERY
 PREHOOK: Input: default@srcpart
 PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=11
 PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=12
  A masked pattern was here 
-POSTHOOK: query: select * from srcpart where ds = "2008-04-08" limit 10
+POSTHOOK: query: select * from srcpart where ds = "2008-04-08" order by key 
limit 10
 POSTHOOK: type: QUERY
 POSTHOOK: Input: default@srcpart
 POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=11
 POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12
  A masked pattern was here 
-238val_238 2008-04-08  11
-86 val_86  2008-04-08  11
-311val_311 2008-04-08  11
-27 val_27  2008-04-08  11
-165val_165 2008-04-08  11
-409val_409 2008-04-08  11
-255val_255 2008-04-08  11
-278val_278 2008-04-08  11
-98 val_98  2008-04-08  11
-484val_484 2008-04-08  11
-PREHOOK: query: select key, count(*) from src group by key limit 10
+0  val_0   2008-04-08  11
+0  val_0   2008

[9/9] hive git commit: HIVE-17743: Add InterfaceAudience and InterfaceStability annotations for Thrift generated APIs (Sahil Takiar, reviewed by Aihua Xu)

2017-10-25 Thread stakiar

HIVE-17743: Add InterfaceAudience and InterfaceStability annotations for Thrift 
generated APIs (Sahil Takiar, reviewed by Aihua Xu)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/b3043a37
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/b3043a37
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/b3043a37

Branch: refs/heads/master
Commit: b3043a37df7906f9cdbd61519a38cd94985f298b
Parents: 18fe39a
Author: Sahil Takiar 
Authored: Wed Oct 25 20:14:28 2017 -0700
Committer: Sahil Takiar 
Committed: Wed Oct 25 20:15:53 2017 -0700

--
 classification/pom.xml  |   73 +
 common/pom.xml  |5 +
 .../classification/InterfaceAudience.java   |   45 -
 .../classification/InterfaceStability.java  |   46 -
 pom.xml |1 +
 service-rpc/pom.xml |   51 +
 .../service/rpc/thrift/TArrayTypeEntry.java |2 +-
 .../hive/service/rpc/thrift/TBinaryColumn.java  |2 +-
 .../hive/service/rpc/thrift/TBoolColumn.java|2 +-
 .../hive/service/rpc/thrift/TBoolValue.java |2 +-
 .../hive/service/rpc/thrift/TByteColumn.java|2 +-
 .../hive/service/rpc/thrift/TByteValue.java |2 +-
 .../hive/service/rpc/thrift/TCLIService.java|  238 +--
 .../rpc/thrift/TCLIServiceConstants.java|2 +-
 .../rpc/thrift/TCancelDelegationTokenReq.java   |2 +-
 .../rpc/thrift/TCancelDelegationTokenResp.java  |2 +-
 .../service/rpc/thrift/TCancelOperationReq.java |2 +-
 .../rpc/thrift/TCancelOperationResp.java|2 +-
 .../service/rpc/thrift/TCloseOperationReq.java  |2 +-
 .../service/rpc/thrift/TCloseOperationResp.java |2 +-
 .../service/rpc/thrift/TCloseSessionReq.java|2 +-
 .../service/rpc/thrift/TCloseSessionResp.java   |2 +-
 .../apache/hive/service/rpc/thrift/TColumn.java |2 +-
 .../hive/service/rpc/thrift/TColumnDesc.java|2 +-
 .../hive/service/rpc/thrift/TColumnValue.java   |2 +-
 .../hive/service/rpc/thrift/TDoubleColumn.java  |2 +-
 .../hive/service/rpc/thrift/TDoubleValue.java   |2 +-
 .../rpc/thrift/TExecuteStatementReq.java|2 +-
 .../rpc/thrift/TExecuteStatementResp.java   |2 +-
 .../service/rpc/thrift/TFetchResultsReq.java|2 +-
 .../service/rpc/thrift/TFetchResultsResp.java   |2 +-
 .../service/rpc/thrift/TGetCatalogsReq.java |2 +-
 .../service/rpc/thrift/TGetCatalogsResp.java|2 +-
 .../hive/service/rpc/thrift/TGetColumnsReq.java |2 +-
 .../service/rpc/thrift/TGetColumnsResp.java |2 +-
 .../rpc/thrift/TGetCrossReferenceReq.java   |2 +-
 .../rpc/thrift/TGetCrossReferenceResp.java  |2 +-
 .../rpc/thrift/TGetDelegationTokenReq.java  |2 +-
 .../rpc/thrift/TGetDelegationTokenResp.java |2 +-
 .../service/rpc/thrift/TGetFunctionsReq.java|2 +-
 .../service/rpc/thrift/TGetFunctionsResp.java   |2 +-
 .../hive/service/rpc/thrift/TGetInfoReq.java|2 +-
 .../hive/service/rpc/thrift/TGetInfoResp.java   |2 +-
 .../hive/service/rpc/thrift/TGetInfoValue.java  |2 +-
 .../rpc/thrift/TGetOperationStatusReq.java  |2 +-
 .../rpc/thrift/TGetOperationStatusResp.java |2 +-
 .../service/rpc/thrift/TGetPrimaryKeysReq.java  |2 +-
 .../service/rpc/thrift/TGetPrimaryKeysResp.java |2 +-
 .../hive/service/rpc/thrift/TGetQueryIdReq.java |2 +-
 .../service/rpc/thrift/TGetQueryIdResp.java |2 +-
 .../rpc/thrift/TGetResultSetMetadataReq.java|2 +-
 .../rpc/thrift/TGetResultSetMetadataResp.java   |2 +-
 .../hive/service/rpc/thrift/TGetSchemasReq.java |2 +-
 .../service/rpc/thrift/TGetSchemasResp.java |2 +-
 .../service/rpc/thrift/TGetTableTypesReq.java   |2 +-
 .../service/rpc/thrift/TGetTableTypesResp.java  |2 +-
 .../hive/service/rpc/thrift/TGetTablesReq.java  |2 +-
 .../hive/service/rpc/thrift/TGetTablesResp.java |2 +-
 .../service/rpc/thrift/TGetTypeInfoReq.java |2 +-
 .../service/rpc/thrift/TGetTypeInfoResp.java|2 +-
 .../service/rpc/thrift/THandleIdentifier.java   |2 +-
 .../hive/service/rpc/thrift/TI16Column.java |2 +-
 .../hive/service/rpc/thrift/TI16Value.java  |2 +-
 .../hive/service/rpc/thrift/TI32Column.java |2 +-
 .../hive/service/rpc/thrift/TI32Value.java  |2 +-
 .../hive/service/rpc/thrift/TI64Column.java |2 +-
 .../hive/service/rpc/thrift/TI64Value.java  |2 +-
 .../hive/service/rpc/thrift/TMapTypeEntry.java  |2 +-
 .../service/rpc/thrift/TOpenSessionReq.java |2 +-
 .../service/rpc/thrift/TOpenSessionResp.java|2 +-
 .../service/rpc/thrift/TOperationHandle.java|2 +-
 .../service/rpc/thrift/TPrimitiveTypeEntry.java |2 +-
 .../service/rpc/thrift/TProgressUpdateResp.java |2 +-
 .../rpc/thri

[1/9] hive git commit: HIVE-17743: Add InterfaceAudience and InterfaceStability annotations for Thrift generated APIs (Sahil Takiar, reviewed by Aihua Xu)

2017-10-25 Thread stakiar

Repository: hive
Updated Branches:
  refs/heads/master 18fe39a70 -> b3043a37d


http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnAbortedException.java
--
diff --git 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnAbortedException.java
 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnAbortedException.java
index 689f646..6947c79 100644
--- 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnAbortedException.java
+++ 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnAbortedException.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class TxnAbortedException extends TException implements 
org.apache.thrift.TBase, 
java.io.Serializable, Cloneable, Comparable {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public class 
TxnAbortedException extends TException implements 
org.apache.thrift.TBase, 
java.io.Serializable, Cloneable, Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("TxnAbortedException");
 
   private static final org.apache.thrift.protocol.TField MESSAGE_FIELD_DESC = 
new org.apache.thrift.protocol.TField("message", 
org.apache.thrift.protocol.TType.STRING, (short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnInfo.java
--
diff --git 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnInfo.java
 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnInfo.java
index ed382ed..dca055e 100644
--- 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnInfo.java
+++ 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnInfo.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class TxnInfo implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public class 
TxnInfo implements org.apache.thrift.TBase, 
java.io.Serializable, Cloneable, Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("TxnInfo");
 
   private static final org.apache.thrift.protocol.TField ID_FIELD_DESC = new 
org.apache.thrift.protocol.TField("id", org.apache.thrift.protocol.TType.I64, 
(short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnOpenException.java
--
diff --git 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnOpenException.java
 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnOpenException.java
index 10b5f9b..229a05a 100644
--- 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnOpenException.java
+++ 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnOpenException.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class TxnOpenException extends TException implements 
org.apache.thrift.TBase, 
java.io.Serializable, Cloneable, Comparable {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public class 
TxnOpenException extends TException implements 
org.apache.thrift.TBase, 
java.io.Serializable, Cloneable, Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("TxnOpenException");
 
   private static final org.apache.thrift.protocol.TField MESSAGE_FIELD_DESC = 
new org.apache.thrift.protocol.TField("message", 
org.apache.thrift.protocol.TType.STRING, (short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Type.java
--

[6/9] hive git commit: HIVE-17743: Add InterfaceAudience and InterfaceStability annotations for Thrift generated APIs (Sahil Takiar, reviewed by Aihua Xu)

2017-10-25 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TMapTypeEntry.java
--
diff --git 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TMapTypeEntry.java
 
b/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TMapTypeEntry.java
index 7ebc15c..8654db6 100644
--- 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TMapTypeEntry.java
+++ 
b/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TMapTypeEntry.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class TMapTypeEntry implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
+@org.apache.hadoop.hive.common.classification.InterfaceAudience.Public 
@org.apache.hadoop.hive.common.classification.InterfaceStability.Stable public 
class TMapTypeEntry implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("TMapTypeEntry");
 
   private static final org.apache.thrift.protocol.TField 
KEY_TYPE_PTR_FIELD_DESC = new org.apache.thrift.protocol.TField("keyTypePtr", 
org.apache.thrift.protocol.TType.I32, (short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq.java
--
diff --git 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq.java
 
b/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq.java
index e47abbb..3195e70 100644
--- 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq.java
+++ 
b/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class TOpenSessionReq implements 
org.apache.thrift.TBase, 
java.io.Serializable, Cloneable, Comparable {
+@org.apache.hadoop.hive.common.classification.InterfaceAudience.Public 
@org.apache.hadoop.hive.common.classification.InterfaceStability.Stable public 
class TOpenSessionReq implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("TOpenSessionReq");
 
   private static final org.apache.thrift.protocol.TField 
CLIENT_PROTOCOL_FIELD_DESC = new 
org.apache.thrift.protocol.TField("client_protocol", 
org.apache.thrift.protocol.TType.I32, (short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionResp.java
--
diff --git 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionResp.java
 
b/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionResp.java
index ee1c87b..c738ee0 100644
--- 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionResp.java
+++ 
b/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionResp.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class TOpenSessionResp implements 
org.apache.thrift.TBase, 
java.io.Serializable, Cloneable, Comparable {
+@org.apache.hadoop.hive.common.classification.InterfaceAudience.Public 
@org.apache.hadoop.hive.common.classification.InterfaceStability.Stable public 
class TOpenSessionResp implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("TOpenSessionResp");
 
   private static final org.apache.thrift.protocol.TField STATUS_FIELD_DESC = 
new org.apache.thrift.protocol.TField("status", 
org.apache.thrift.protocol.TType.STRUCT, (short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOperationHandle.java
--
diff --git 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOperationHandle.java
 
b/service-rpc/src/gen/thrift/

[4/9] hive git commit: HIVE-17743: Add InterfaceAudience and InterfaceStability annotations for Thrift generated APIs (Sahil Takiar, reviewed by Aihua Xu)

2017-10-25 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockComponent.java
--
diff --git 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockComponent.java
 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockComponent.java
index d349508..0307540 100644
--- 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockComponent.java
+++ 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockComponent.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class LockComponent implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public class 
LockComponent implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("LockComponent");
 
   private static final org.apache.thrift.protocol.TField TYPE_FIELD_DESC = new 
org.apache.thrift.protocol.TField("type", org.apache.thrift.protocol.TType.I32, 
(short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockRequest.java
--
diff --git 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockRequest.java
 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockRequest.java
index 75d9d56..6aaed5c 100644
--- 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockRequest.java
+++ 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockRequest.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class LockRequest implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public class 
LockRequest implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("LockRequest");
 
   private static final org.apache.thrift.protocol.TField COMPONENT_FIELD_DESC 
= new org.apache.thrift.protocol.TField("component", 
org.apache.thrift.protocol.TType.LIST, (short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockResponse.java
--
diff --git 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockResponse.java
 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockResponse.java
index 862c67f..fdaab4b 100644
--- 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockResponse.java
+++ 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LockResponse.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class LockResponse implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public class 
LockResponse implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("LockResponse");
 
   private static final org.apache.thrift.protocol.TField LOCKID_FIELD_DESC = 
new org.apache.thrift.protocol.TField("lockid", 
org.apache.thrift.protocol.TType.I64, (short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LongColumnStatsData.java
--
diff --git 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/LongColumnStatsData.java
 
b/sta

[5/9] hive git commit: HIVE-17743: Add InterfaceAudience and InterfaceStability annotations for Thrift generated APIs (Sahil Takiar, reviewed by Aihua Xu)

2017-10-25 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ConfigValSecurityException.java
--
diff --git 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ConfigValSecurityException.java
 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ConfigValSecurityException.java
index d454a96..c4837a1 100644
--- 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ConfigValSecurityException.java
+++ 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ConfigValSecurityException.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class ConfigValSecurityException extends TException implements 
org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public class 
ConfigValSecurityException extends TException implements 
org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("ConfigValSecurityException");
 
   private static final org.apache.thrift.protocol.TField MESSAGE_FIELD_DESC = 
new org.apache.thrift.protocol.TField("message", 
org.apache.thrift.protocol.TType.STRING, (short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CurrentNotificationEventId.java
--
diff --git 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CurrentNotificationEventId.java
 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CurrentNotificationEventId.java
index 2565a08..536829f 100644
--- 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CurrentNotificationEventId.java
+++ 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CurrentNotificationEventId.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class CurrentNotificationEventId implements 
org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public class 
CurrentNotificationEventId implements 
org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("CurrentNotificationEventId");
 
   private static final org.apache.thrift.protocol.TField EVENT_ID_FIELD_DESC = 
new org.apache.thrift.protocol.TField("eventId", 
org.apache.thrift.protocol.TType.I64, (short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
--
diff --git 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
index 2769845..1a8c7b5 100644
--- 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
+++ 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class Database implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, Comparable {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public class 
Database implements org.apache.thrift.TBase, 
java.io.Serializable, Cloneable, Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("Database");
 
   private static final org.apache.thrift.protocol.TField NAME_FIELD_DESC = new 
org.apache.thrift.protocol.TField("name", 
org.apache.thrift.protocol.TType.STRING, (short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/met

[3/9] hive git commit: HIVE-17743: Add InterfaceAudience and InterfaceStability annotations for Thrift generated APIs (Sahil Takiar, reviewed by Aihua Xu)

2017-10-25 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksRequest.java
--
diff --git 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksRequest.java
 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksRequest.java
index f566d6f..9fbab42 100644
--- 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksRequest.java
+++ 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksRequest.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class ShowLocksRequest implements 
org.apache.thrift.TBase, 
java.io.Serializable, Cloneable, Comparable {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public class 
ShowLocksRequest implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("ShowLocksRequest");
 
   private static final org.apache.thrift.protocol.TField DBNAME_FIELD_DESC = 
new org.apache.thrift.protocol.TField("dbname", 
org.apache.thrift.protocol.TType.STRING, (short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponse.java
--
diff --git 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponse.java
 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponse.java
index 0fe5812..f22deb2 100644
--- 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponse.java
+++ 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponse.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class ShowLocksResponse implements 
org.apache.thrift.TBase, 
java.io.Serializable, Cloneable, Comparable {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public class 
ShowLocksResponse implements org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("ShowLocksResponse");
 
   private static final org.apache.thrift.protocol.TField LOCKS_FIELD_DESC = 
new org.apache.thrift.protocol.TField("locks", 
org.apache.thrift.protocol.TType.LIST, (short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponseElement.java
--
diff --git 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponseElement.java
 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponseElement.java
index 4bd7a7e..13df2bf 100644
--- 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponseElement.java
+++ 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowLocksResponseElement.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class ShowLocksResponseElement implements 
org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public class 
ShowLocksResponseElement implements 
org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("ShowLocksResponseElement");
 
   private static final org.apache.thrift.protocol.TField LOCKID_FIELD_DESC = 
new org.apache.thrift.protocol.TField("lockid", 
org.apache.thrift.protocol.TType.I64, (short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java
---

[8/9] hive git commit: HIVE-17743: Add InterfaceAudience and InterfaceStability annotations for Thrift generated APIs (Sahil Takiar, reviewed by Aihua Xu)

2017-10-25 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIService.java
--
diff --git 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIService.java
 
b/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIService.java
index 2f44a4f..b6ed40b 100644
--- 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIService.java
+++ 
b/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIService.java
@@ -35,9 +35,9 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class TCLIService {
+@org.apache.hadoop.hive.common.classification.InterfaceAudience.Public 
@org.apache.hadoop.hive.common.classification.InterfaceStability.Stable public 
class TCLIService {
 
-  public interface Iface {
+  @org.apache.hadoop.hive.common.classification.InterfaceAudience.Public 
@org.apache.hadoop.hive.common.classification.InterfaceStability.Stable public 
interface Iface {
 
 public TOpenSessionResp OpenSession(TOpenSessionReq req) throws 
org.apache.thrift.TException;
 
@@ -85,7 +85,7 @@ public class TCLIService {
 
   }
 
-  public interface AsyncIface {
+  @org.apache.hadoop.hive.common.classification.InterfaceAudience.Public 
@org.apache.hadoop.hive.common.classification.InterfaceStability.Stable public 
interface AsyncIface {
 
 public void OpenSession(TOpenSessionReq req, 
org.apache.thrift.async.AsyncMethodCallback resultHandler) throws 
org.apache.thrift.TException;
 
@@ -133,8 +133,8 @@ public class TCLIService {
 
   }
 
-  public static class Client extends org.apache.thrift.TServiceClient 
implements Iface {
-public static class Factory implements 
org.apache.thrift.TServiceClientFactory {
+  @org.apache.hadoop.hive.common.classification.InterfaceAudience.Public 
@org.apache.hadoop.hive.common.classification.InterfaceStability.Stable public 
static class Client extends org.apache.thrift.TServiceClient implements Iface {
+@org.apache.hadoop.hive.common.classification.InterfaceAudience.Public 
@org.apache.hadoop.hive.common.classification.InterfaceStability.Stable public 
static class Factory implements org.apache.thrift.TServiceClientFactory 
{
   public Factory() {}
   public Client getClient(org.apache.thrift.protocol.TProtocol prot) {
 return new Client(prot);
@@ -660,8 +660,8 @@ public class TCLIService {
 }
 
   }
-  public static class AsyncClient extends org.apache.thrift.async.TAsyncClient 
implements AsyncIface {
-public static class Factory implements 
org.apache.thrift.async.TAsyncClientFactory {
+  @org.apache.hadoop.hive.common.classification.InterfaceAudience.Public 
@org.apache.hadoop.hive.common.classification.InterfaceStability.Stable public 
static class AsyncClient extends org.apache.thrift.async.TAsyncClient 
implements AsyncIface {
+@org.apache.hadoop.hive.common.classification.InterfaceAudience.Public 
@org.apache.hadoop.hive.common.classification.InterfaceStability.Stable public 
static class Factory implements 
org.apache.thrift.async.TAsyncClientFactory {
   private org.apache.thrift.async.TAsyncClientManager clientManager;
   private org.apache.thrift.protocol.TProtocolFactory protocolFactory;
   public Factory(org.apache.thrift.async.TAsyncClientManager 
clientManager, org.apache.thrift.protocol.TProtocolFactory protocolFactory) {
@@ -684,7 +684,7 @@ public class TCLIService {
   ___manager.call(method_call);
 }
 
-public static class OpenSession_call extends 
org.apache.thrift.async.TAsyncMethodCall {
+@org.apache.hadoop.hive.common.classification.InterfaceAudience.Public 
@org.apache.hadoop.hive.common.classification.InterfaceStability.Stable public 
static class OpenSession_call extends org.apache.thrift.async.TAsyncMethodCall {
   private TOpenSessionReq req;
   public OpenSession_call(TOpenSessionReq req, 
org.apache.thrift.async.AsyncMethodCallback resultHandler, 
org.apache.thrift.async.TAsyncClient client, 
org.apache.thrift.protocol.TProtocolFactory protocolFactory, 
org.apache.thrift.transport.TNonblockingTransport transport) throws 
org.apache.thrift.TException {
 super(client, protocolFactory, transport, resultHandler, false);
@@ -716,7 +716,7 @@ public class TCLIService {
   ___manager.call(method_call);
 }
 
-public static class CloseSession_call extends 
org.apache.thrift.async.TAsyncMethodCall {
+@org.apache.hadoop.hive.common.classification.InterfaceAudience.Public 
@org.apache.hadoop.hive.common.classification.InterfaceStability.Stable public 
static class CloseSession_call extends org.apache.thrift.async.TAsyncMethodCall 
{
   private TCloseSessionReq req;
   public CloseSession_call(TCloseSessionR

[7/9] hive git commit: HIVE-17743: Add InterfaceAudience and InterfaceStability annotations for Thrift generated APIs (Sahil Takiar, reviewed by Aihua Xu)

2017-10-25 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java
--
diff --git 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java
 
b/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java
index 955da0c..e054aa6 100644
--- 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java
+++ 
b/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java
@@ -34,7 +34,7 @@ import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
-public class TCLIServiceConstants {
+@org.apache.hadoop.hive.common.classification.InterfaceAudience.Public 
@org.apache.hadoop.hive.common.classification.InterfaceStability.Stable public 
class TCLIServiceConstants {
 
   public static final Set PRIMITIVE_TYPES = new HashSet();
   static {

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCancelDelegationTokenReq.java
--
diff --git 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCancelDelegationTokenReq.java
 
b/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCancelDelegationTokenReq.java
index a7d4e7d..6c1da6b 100644
--- 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCancelDelegationTokenReq.java
+++ 
b/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCancelDelegationTokenReq.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class TCancelDelegationTokenReq implements 
org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
+@org.apache.hadoop.hive.common.classification.InterfaceAudience.Public 
@org.apache.hadoop.hive.common.classification.InterfaceStability.Stable public 
class TCancelDelegationTokenReq implements 
org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("TCancelDelegationTokenReq");
 
   private static final org.apache.thrift.protocol.TField 
SESSION_HANDLE_FIELD_DESC = new 
org.apache.thrift.protocol.TField("sessionHandle", 
org.apache.thrift.protocol.TType.STRUCT, (short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCancelDelegationTokenResp.java
--
diff --git 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCancelDelegationTokenResp.java
 
b/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCancelDelegationTokenResp.java
index 611e92c..689bc7a 100644
--- 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCancelDelegationTokenResp.java
+++ 
b/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCancelDelegationTokenResp.java
@@ -35,7 +35,7 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class TCancelDelegationTokenResp implements 
org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
+@org.apache.hadoop.hive.common.classification.InterfaceAudience.Public 
@org.apache.hadoop.hive.common.classification.InterfaceStability.Stable public 
class TCancelDelegationTokenResp implements 
org.apache.thrift.TBase, java.io.Serializable, Cloneable, 
Comparable {
   private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new 
org.apache.thrift.protocol.TStruct("TCancelDelegationTokenResp");
 
   private static final org.apache.thrift.protocol.TField STATUS_FIELD_DESC = 
new org.apache.thrift.protocol.TField("status", 
org.apache.thrift.protocol.TType.STRUCT, (short)1);

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCancelOperationReq.java
--
diff --git 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCancelOperationReq.java
 
b/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCancelOperationReq.java
index 4076c57..91a468c 100644
--- 
a/service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCancelOperationReq.java
+++ 
b/service-rpc/

[2/9] hive git commit: HIVE-17743: Add InterfaceAudience and InterfaceStability annotations for Thrift generated APIs (Sahil Takiar, reviewed by Aihua Xu)

2017-10-25 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/b3043a37/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
--
diff --git 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
index 53b47e6..48bfb05 100644
--- 
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
+++ 
b/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
@@ -35,12 +35,12 @@ import org.slf4j.LoggerFactory;
 
 @SuppressWarnings({"cast", "rawtypes", "serial", "unchecked"})
 @Generated(value = "Autogenerated by Thrift Compiler (0.9.3)")
-public class ThriftHiveMetastore {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public class 
ThriftHiveMetastore {
 
   /**
* This interface is live.
*/
-  public interface Iface extends com.facebook.fb303.FacebookService.Iface {
+  @org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public interface 
Iface extends com.facebook.fb303.FacebookService.Iface {
 
 public String getMetaConf(String key) throws MetaException, 
org.apache.thrift.TException;
 
@@ -380,7 +380,7 @@ public class ThriftHiveMetastore {
 
   }
 
-  public interface AsyncIface extends com.facebook.fb303.FacebookService 
.AsyncIface {
+  @org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public interface 
AsyncIface extends com.facebook.fb303.FacebookService .AsyncIface {
 
 public void getMetaConf(String key, 
org.apache.thrift.async.AsyncMethodCallback resultHandler) throws 
org.apache.thrift.TException;
 
@@ -720,8 +720,8 @@ public class ThriftHiveMetastore {
 
   }
 
-  public static class Client extends com.facebook.fb303.FacebookService.Client 
implements Iface {
-public static class Factory implements 
org.apache.thrift.TServiceClientFactory {
+  @org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public static class 
Client extends com.facebook.fb303.FacebookService.Client implements Iface {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public static class 
Factory implements org.apache.thrift.TServiceClientFactory {
   public Factory() {}
   public Client getClient(org.apache.thrift.protocol.TProtocol prot) {
 return new Client(prot);
@@ -5566,8 +5566,8 @@ public class ThriftHiveMetastore {
 }
 
   }
-  public static class AsyncClient extends 
com.facebook.fb303.FacebookService.AsyncClient implements AsyncIface {
-public static class Factory implements 
org.apache.thrift.async.TAsyncClientFactory {
+  @org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public static class 
AsyncClient extends com.facebook.fb303.FacebookService.AsyncClient implements 
AsyncIface {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public static class 
Factory implements org.apache.thrift.async.TAsyncClientFactory {
   private org.apache.thrift.async.TAsyncClientManager clientManager;
   private org.apache.thrift.protocol.TProtocolFactory protocolFactory;
   public Factory(org.apache.thrift.async.TAsyncClientManager 
clientManager, org.apache.thrift.protocol.TProtocolFactory protocolFactory) {
@@ -5590,7 +5590,7 @@ public class ThriftHiveMetastore {
   ___manager.call(method_call);
 }
 
-public static class getMetaConf_call extends 
org.apache.thrift.async.TAsyncMethodCall {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classification.InterfaceStability.Stable public static class 
getMetaConf_call extends org.apache.thrift.async.TAsyncMethodCall {
   private String key;
   public getMetaConf_call(String key, 
org.apache.thrift.async.AsyncMethodCallback resultHandler, 
org.apache.thrift.async.TAsyncClient client, 
org.apache.thrift.protocol.TProtocolFactory protocolFactory, 
org.apache.thrift.transport.TNonblockingTransport transport) throws 
org.apache.thrift.TException {
 super(client, protocolFactory, transport, resultHandler, false);
@@ -5622,7 +5622,7 @@ public class ThriftHiveMetastore {
   ___manager.call(method_call);
 }
 
-public static class setMetaConf_call extends 
org.apache.thrift.async.TAsyncMethodCall {
+@org.apache.hadoop.classification.InterfaceAudience.Public 
@org.apache.hadoop.classificatio

hive git commit: HIVE-16601: Display Session Id and Query Name / Id in Spark UI (Sahil Takiar, reviewed by Barna Zsombor Klara, Peter Vary, Xuefu Zhang)

2017-10-24 Thread stakiar

Repository: hive
Updated Branches:
  refs/heads/master 152ee221e -> a284df1f8


HIVE-16601: Display Session Id and Query Name / Id in Spark UI (Sahil Takiar, 
reviewed by Barna Zsombor Klara, Peter Vary, Xuefu Zhang)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/a284df1f
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/a284df1f
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/a284df1f

Branch: refs/heads/master
Commit: a284df1f87eccee8bdad04afea2150e6c07337a0
Parents: 152ee22
Author: Sahil Takiar 
Authored: Tue Oct 24 08:27:24 2017 -0700
Committer: Sahil Takiar 
Committed: Tue Oct 24 08:27:24 2017 -0700

--
 .../org/apache/hadoop/hive/conf/HiveConf.java   |  3 +-
 .../java/org/apache/hadoop/hive/ql/Driver.java  |  6 ++-
 .../apache/hadoop/hive/ql/exec/DagUtils.java| 39 
 .../ql/exec/spark/HiveSparkClientFactory.java   | 16 ++--
 .../ql/exec/spark/RemoteHiveSparkClient.java|  3 ++
 .../ql/exec/spark/session/SparkSessionImpl.java |  2 +-
 .../spark/session/SparkSessionManagerImpl.java  |  2 +-
 .../apache/hadoop/hive/ql/plan/SparkWork.java   | 15 +---
 ql/src/test/queries/clientpositive/parallel.q   |  2 +-
 9 files changed, 73 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/a284df1f/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
--
diff --git a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
index a6ecb37..62dcbd5 100644
--- a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
+++ b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
@@ -1510,7 +1510,8 @@ public class HiveConf extends Configuration {
 "Whether to grant access to the hs2/hive user for queries"),
 HIVEQUERYNAME ("hive.query.name", null,
 "This named is used by Tez to set the dag name. This name in turn will 
appear on \n" +
-"the Tez UI representing the work that was done."),
+"the Tez UI representing the work that was done. Used by Spark to set 
the query name, will show up in the\n" +
+"Spark UI."),
 
 HIVEOPTIMIZEBUCKETINGSORTING("hive.optimize.bucketingsorting", true,
 "Don't create a reducer for enforcing \n" +

http://git-wip-us.apache.org/repos/asf/hive/blob/a284df1f/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
--
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 
b/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
index b7f76ab..6c6ad92 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
@@ -132,6 +132,8 @@ import com.google.common.collect.Sets;
 
 public class Driver implements CommandProcessor {
 
+  public static final String MAPREDUCE_WORKFLOW_NODE_NAME = 
"mapreduce.workflow.node.name";
+
   static final private String CLASS_NAME = Driver.class.getName();
   private static final Logger LOG = LoggerFactory.getLogger(CLASS_NAME);
   static final private LogHelper console = new LogHelper(LOG);
@@ -2248,9 +2250,9 @@ public class Driver implements CommandProcessor {
 }
 if (tsk.isMapRedTask() && !(tsk instanceof ConditionalTask)) {
   if (noName) {
-conf.set(MRJobConfig.JOB_NAME, jobname + "(" + tsk.getId() + ")");
+conf.set(MRJobConfig.JOB_NAME, jobname + " (" + tsk.getId() + ")");
   }
-  conf.set("mapreduce.workflow.node.name", tsk.getId());
+  conf.set(MAPREDUCE_WORKFLOW_NODE_NAME, tsk.getId());
   Utilities.setWorkflowAdjacencies(conf, plan);
   cxt.incCurJobNo(1);
   console.printInfo("Launching Job " + cxt.getCurJobNo() + " out of " + 
jobs);

http://git-wip-us.apache.org/repos/asf/hive/blob/a284df1f/ql/src/java/org/apache/hadoop/hive/ql/exec/DagUtils.java
--
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/DagUtils.java 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/DagUtils.java
new file mode 100644
index 000..aed1b2c
--- /dev/null
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/DagUtils.java
@@ -0,0 +1,39 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in

hive git commit: HIVE-17789: Flaky test: TestSessionManagerMetrics.testAbandonedSessionMetrics has timing related problems (Andrew Sherman, reviewed by Sahil Takiar, Aihua Xu)

2017-10-18 Thread stakiar

Repository: hive
Updated Branches:
  refs/heads/master c129bb95d -> c6c374eb0


HIVE-17789: Flaky test: TestSessionManagerMetrics.testAbandonedSessionMetrics 
has timing related problems (Andrew Sherman, reviewed by Sahil Takiar, Aihua Xu)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/c6c374eb
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/c6c374eb
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/c6c374eb

Branch: refs/heads/master
Commit: c6c374eb09c28baa4d8fd9e5de30aa3554ef804e
Parents: c129bb9
Author: Andrew Sherman 
Authored: Wed Oct 18 14:16:07 2017 -0700
Committer: Sahil Takiar 
Committed: Wed Oct 18 14:16:59 2017 -0700

--
 .../cli/session/TestSessionManagerMetrics.java  | 22 +++-
 1 file changed, 17 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/c6c374eb/service/src/test/org/apache/hive/service/cli/session/TestSessionManagerMetrics.java
--
diff --git 
a/service/src/test/org/apache/hive/service/cli/session/TestSessionManagerMetrics.java
 
b/service/src/test/org/apache/hive/service/cli/session/TestSessionManagerMetrics.java
index 5f418c7..646159f 100644
--- 
a/service/src/test/org/apache/hive/service/cli/session/TestSessionManagerMetrics.java
+++ 
b/service/src/test/org/apache/hive/service/cli/session/TestSessionManagerMetrics.java
@@ -25,6 +25,8 @@ import java.util.concurrent.BrokenBarrierException;
 import java.util.concurrent.CyclicBarrier;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
+
+import com.fasterxml.jackson.databind.JsonNode;
 import org.apache.hadoop.hive.common.metrics.MetricsTestUtils;
 import org.apache.hadoop.hive.common.metrics.common.MetricsConstant;
 import org.apache.hadoop.hive.common.metrics.common.MetricsFactory;
@@ -32,7 +34,6 @@ import 
org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics;
 import org.apache.hadoop.hive.common.metrics.metrics2.MetricsReporting;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.ql.metadata.Hive;
-import org.apache.hadoop.util.Time;
 import org.apache.hive.service.cli.FetchOrientation;
 import org.apache.hive.service.cli.HiveSQLException;
 import org.apache.hive.service.cli.OperationHandle;
@@ -373,9 +374,20 @@ public class TestSessionManagerMetrics {
 sm.openSession(TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V9, "user", 
"passw", "127.0.0.1",
 new HashMap());
 
-Thread.sleep(3200);
-
-json = metrics.dumpJson();
-MetricsTestUtils.verifyMetricsJson(json, MetricsTestUtils.COUNTER, 
MetricsConstant.HS2_ABANDONED_SESSIONS, 1);
+// We're going to wait for the session to be abandoned.
+String currentValue;
+int count = 5; // how many times we'll sleep before giving up
+String expectedValue = "1";
+do {
+  // HIVE_SERVER2_SESSION_CHECK_INTERVAL is set to 3 seconds, so we have 
to wait for at least
+  // that long to see an abandoned session
+  Thread.sleep(3200);
+  json = metrics.dumpJson();
+  currentValue = MetricsTestUtils
+  .getJsonNode(json, MetricsTestUtils.COUNTER, 
MetricsConstant.HS2_ABANDONED_SESSIONS)
+  .asText();
+  // loop until the value is correct or we run out of tries
+} while (!expectedValue.equals(currentValue) && --count > 0);
+Assert.assertEquals(expectedValue, currentValue);
   }
 }

hive git commit: HIVE-16395: ConcurrentModificationException on config object in HoS (Andrew Sherman via Sahil Takiar)

2017-10-15 Thread stakiar

Repository: hive
Updated Branches:
  refs/heads/master 133d3c473 -> e33126281


HIVE-16395: ConcurrentModificationException on config object in HoS (Andrew 
Sherman via Sahil Takiar)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/e3312628
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/e3312628
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/e3312628

Branch: refs/heads/master
Commit: e331262813027ca2a1aae7fedcd1c8863ed6b751
Parents: 133d3c4
Author: Andrew Sherman 
Authored: Sun Oct 15 17:16:35 2017 -0700
Committer: Sahil Takiar 
Committed: Sun Oct 15 17:16:35 2017 -0700

--
 .../ql/exec/spark/HiveSparkClientFactory.java   |  8 ++-
 .../ql/exec/spark/session/SparkSessionImpl.java |  6 +++
 .../session/TestSparkSessionManagerImpl.java| 51 
 3 files changed, 64 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/e3312628/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java
--
diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java
index 194585e..597fcab 100644
--- 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java
+++ 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java
@@ -26,10 +26,10 @@ import java.util.Map;
 import java.util.Properties;
 import java.util.Set;
 
+import com.google.common.annotations.VisibleForTesting;
 import org.apache.commons.compress.utils.CharsetNames;
 import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
 import org.apache.hadoop.hive.common.LogUtils;
-import org.apache.hadoop.hive.ql.exec.Utilities;
 import org.apache.hadoop.hive.ql.session.SessionState;
 import org.apache.hive.spark.client.SparkClientUtilities;
 import org.slf4j.Logger;
@@ -60,6 +60,8 @@ public class HiveSparkClientFactory {
   private static final String SPARK_DEFAULT_REFERENCE_TRACKING = "false";
   private static final String SPARK_WAIT_APP_COMPLETE = 
"spark.yarn.submit.waitAppCompletion";
   private static final String SPARK_DEPLOY_MODE = "spark.submit.deployMode";
+  @VisibleForTesting
+  public static final String SPARK_CLONE_CONFIGURATION = 
"spark.hadoop.cloneConf";
 
   public static HiveSparkClient createHiveSparkClient(HiveConf hiveconf) 
throws Exception {
 Map sparkConf = initiateSparkConf(hiveconf);
@@ -222,6 +224,10 @@ public class HiveSparkClientFactory {
   sparkConf.put(SPARK_WAIT_APP_COMPLETE, "false");
 }
 
+// Force Spark configs to be cloned by default
+sparkConf.putIfAbsent(SPARK_CLONE_CONFIGURATION, "true");
+
+
 // Set the credential provider passwords if found, if there is job 
specific password
 // the credential provider location is set directly in the execute method 
of LocalSparkClient
 // and submit method of RemoteHiveSparkClient when the job config is 
created

http://git-wip-us.apache.org/repos/asf/hive/blob/e3312628/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java
--
diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java
 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java
index 54d2cec..8d79dd9 100644
--- 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java
+++ 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.hive.ql.exec.spark.session;
 import java.io.IOException;
 import java.util.UUID;
 
+import com.google.common.annotations.VisibleForTesting;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.fs.permission.FsPermission;
@@ -174,4 +175,9 @@ public class SparkSessionImpl implements SparkSession {
   public static String makeSessionId() {
 return UUID.randomUUID().toString();
   }
+
+  @VisibleForTesting
+  HiveSparkClient getHiveSparkClient() {
+return hiveSparkClient;
+  }
 }

http://git-wip-us.apache.org/repos/asf/hive/blob/e3312628/ql/src/test/org/apache/hadoop/hive/ql/exec/spark/session/TestSparkSessionManagerImpl.java
--
diff --git 
a/ql/src/test/org/apache/hadoop/hive/ql/exec/spark/session/TestSparkSessionManagerImpl.java
 
b/ql/src/test/org/apache/hadoop/hive/ql/exec/spark/session/TestSparkSessionManagerImpl.java
index 489383b..47d2437 100644
--- 
a/ql/src/test/org/apache/hadoop/hive/ql/exec/spark/session/TestSparkSessionManagerImpl.java
+++ 
b/ql/src/test/org/apache/hadoop/hive/ql/exec/spark/session/TestSparkSe

hive git commit: HIVE-17111: Add TestLocalSparkCliDriver (Sahil Takiar, reviewed by Aihua Xu, Peter Vary, Xuefu Zhang) (addendum)

2017-10-10 Thread stakiar

Repository: hive
Updated Branches:
  refs/heads/master a974a9e6c -> 857347f7f


HIVE-17111: Add TestLocalSparkCliDriver (Sahil Takiar, reviewed by Aihua Xu, 
Peter Vary, Xuefu Zhang) (addendum)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/857347f7
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/857347f7
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/857347f7

Branch: refs/heads/master
Commit: 857347f7fde23d9fb2a0026ca8f1ab6faa28b91c
Parents: a974a9e
Author: Sahil Takiar 
Authored: Tue Oct 10 13:04:11 2017 -0700
Committer: Sahil Takiar 
Committed: Tue Oct 10 13:04:11 2017 -0700

--
 .../main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java| 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/857347f7/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
--
diff --git 
a/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
b/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
index c9e1543..438d296 100644
--- 
a/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
+++ 
b/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
@@ -54,6 +54,7 @@ public class CliConfigs {
 excludesFrom(testConfigProps, "spark.only.query.files");
 excludesFrom(testConfigProps, "miniSparkOnYarn.only.query.files");
 excludesFrom(testConfigProps, "disabled.query.files");
+excludesFrom(testConfigProps, "localSpark.only.query.files");
 
 setResultsDir("ql/src/test/results/clientpositive");
 setLogDir("itests/qtest/target/qfile-results/clientpositive");

hive git commit: HIVE-17111: Add TestLocalSparkCliDriver (Sahil Takiar, reviewed by Aihua Xu, Peter Vary, Xuefu Zhang)

2017-10-09 Thread stakiar

Repository: hive
Updated Branches:
  refs/heads/master 842d4dfc8 -> c25455746


HIVE-17111: Add TestLocalSparkCliDriver (Sahil Takiar, reviewed by Aihua Xu, 
Peter Vary, Xuefu Zhang)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/c2545574
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/c2545574
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/c2545574

Branch: refs/heads/master
Commit: c25455746ae46af61e44591ba3ee4833f0b4b8d0
Parents: 842d4df
Author: Sahil Takiar 
Authored: Mon Oct 9 16:05:16 2017 -0700
Committer: Sahil Takiar 
Committed: Mon Oct 9 16:05:16 2017 -0700

--
 data/conf/spark/local/hive-site.xml | 258 +++
 .../hive/cli/TestLocalSparkCliDriver.java   |  62 +
 .../test/resources/testconfiguration.properties |   1 +
 .../hadoop/hive/cli/control/CliConfigs.java |  22 ++
 pom.xml |   2 +
 .../clientpositive/spark_local_queries.q|  16 ++
 .../spark/spark_local_queries.q.out | 131 ++
 7 files changed, 492 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/c2545574/data/conf/spark/local/hive-site.xml
--
diff --git a/data/conf/spark/local/hive-site.xml 
b/data/conf/spark/local/hive-site.xml
new file mode 100644
index 000..8bade0f
--- /dev/null
+++ b/data/conf/spark/local/hive-site.xml
@@ -0,0 +1,258 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  hadoop.tmp.dir
+  ${test.tmp.dir}/hadoop-tmp
+  A base for other temporary directories.
+
+
+
+  hive.exec.scratchdir
+  ${test.tmp.dir}/scratchdir
+  Scratch space for Hive jobs
+
+
+
+  hive.exec.local.scratchdir
+  ${test.tmp.dir}/localscratchdir/
+  Local scratch space for Hive jobs
+
+
+
+  datanucleus.schema.autoCreateAll
+  true
+
+
+
+  hive.metastore.schema.verification
+  false
+
+
+
+  javax.jdo.option.ConnectionURL
+  
jdbc:derby:;databaseName=${test.tmp.dir}/junit_metastore_db;create=true
+
+
+
+  javax.jdo.option.ConnectionDriverName
+  org.apache.derby.jdbc.EmbeddedDriver
+
+
+
+  javax.jdo.option.ConnectionUserName
+  APP
+
+
+
+  javax.jdo.option.ConnectionPassword
+  mine
+
+
+
+  
+  hive.metastore.warehouse.dir
+  ${test.warehouse.dir}
+  
+
+
+
+  hive.metastore.metadb.dir
+  file://${test.tmp.dir}/metadb/
+  
+  Required by metastore server or if the uris argument below is not supplied
+  
+
+
+
+  test.log.dir
+  ${test.tmp.dir}/log/
+  
+
+
+
+  test.data.files
+  ${hive.root}/data/files
+  
+
+
+
+  hive.jar.path
+  
${maven.local.repository}/org/apache/hive/hive-exec/${hive.version}/hive-exec-${hive.version}.jar
+  
+
+
+
+  test.data.scripts
+  ${hive.root}/data/scripts
+  
+
+
+
+  hive.metastore.rawstore.impl
+  org.apache.hadoop.hive.metastore.ObjectStore
+  Name of the class that implements 
org.apache.hadoop.hive.metastore.rawstore interface. This class is used to 
store and retrieval of raw metadata objects such as table, 
database
+
+
+
+  hive.querylog.location
+  ${test.tmp.dir}/tmp
+  Location of the structured hive logs
+
+
+
+  hive.exec.pre.hooks
+  org.apache.hadoop.hive.ql.hooks.PreExecutePrinter, 
org.apache.hadoop.hive.ql.hooks.EnforceReadOnlyTables
+  Pre Execute Hook for Tests
+
+
+
+  hive.exec.post.hooks
+  org.apache.hadoop.hive.ql.hooks.PostExecutePrinter
+  Post Execute Hook for Tests
+
+
+
+  hive.support.concurrency
+  false
+  Whether hive supports concurrency or not. A zookeeper instance 
must be up and running for the default hive lock manager to support read-write 
locks.
+
+
+
+  fs.pfile.impl
+  org.apache.hadoop.fs.ProxyLocalFileSystem
+  A proxy for local file system used for cross file system 
testing
+
+
+
+  hive.exec.mode.local.auto
+  false
+  
+Let hive determine whether to run in local mode automatically
+Disabling this for tests so that minimr is not affected
+  
+
+
+
+  hive.auto.convert.join
+  false
+  Whether Hive enable the optimization about converting common 
join into mapjoin based on the input file size
+
+
+
+  hive.ignore.mapjoin.hint
+  false
+  Whether Hive ignores the mapjoin hint
+
+
+
+  io.sort.mb
+  10
+
+
+
+  hive.input.format
+  org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
+  The default input format, if it is not specified, the system 
assigns it. It is set to HiveInputFormat for hadoop versions 17, 18 and 19, 
whereas it is set to CombineHiveInputFormat for hadoop 20. The user can always 
overwrite it - if there is a bug in CombineHiveInputFormat, it can always be 
manually set to HiveInputFormat. 
+
+
+
+  hive.default.rcfile.serde
+  org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe
+  The default SerDe hive will use for the rcfile 
format
+
+
+
+  hive.stats.dbclass
+  fs
+  The default storatge that stores temporary hive statistics. 
Currently, fs typ

[3/3] hive git commit: HIVE-13843: Re-enable the HoS tests disabled in HIVE-13402 (Sahil Takiar, reviewed by Rui Li)

2017-10-01 Thread stakiar

HIVE-13843: Re-enable the HoS tests disabled in HIVE-13402 (Sahil Takiar, 
reviewed by Rui Li)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/2948c160
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/2948c160
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/2948c160

Branch: refs/heads/master
Commit: 2948c160fe5b5666cab68f6667549b47f21c1b7a
Parents: 9244fdc
Author: Sahil Takiar 
Authored: Sun Oct 1 19:33:12 2017 -0700
Committer: Sahil Takiar 
Committed: Sun Oct 1 19:33:12 2017 -0700

--
 .../test/resources/testconfiguration.properties |7 +
 .../clientpositive/spark/auto_join32.q.out  |  574 +++
 .../spark/auto_sortmerge_join_2.q.out   |  594 +++
 .../results/clientpositive/spark/insert1.q.out  |  412 ++
 .../results/clientpositive/spark/join40.q.out   | 3845 ++
 .../clientpositive/spark/join_empty.q.out   |   56 +
 .../clientpositive/spark/ppd_join4.q.out|  128 +
 .../clientpositive/spark/transform1.q.out   |  138 +
 8 files changed, 5754 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/2948c160/itests/src/test/resources/testconfiguration.properties
--
diff --git a/itests/src/test/resources/testconfiguration.properties 
b/itests/src/test/resources/testconfiguration.properties
index 2df49a7..038487f 100644
--- a/itests/src/test/resources/testconfiguration.properties
+++ b/itests/src/test/resources/testconfiguration.properties
@@ -842,6 +842,7 @@ spark.query.files=add_part_multiple.q, \
   auto_join3.q, \
   auto_join30.q, \
   auto_join31.q, \
+  auto_join32.q, \
   auto_join4.q, \
   auto_join5.q, \
   auto_join6.q, \
@@ -862,6 +863,7 @@ spark.query.files=add_part_multiple.q, \
   auto_sortmerge_join_14.q, \
   auto_sortmerge_join_15.q, \
   auto_sortmerge_join_16.q, \
+  auto_sortmerge_join_2.q, \
   auto_sortmerge_join_3.q, \
   auto_sortmerge_join_4.q, \
   auto_sortmerge_join_5.q, \
@@ -1008,6 +1010,7 @@ spark.query.files=add_part_multiple.q, \
   input18.q, \
   input1_limit.q, \
   input_part2.q, \
+  insert1.q, \
   insert_into1.q, \
   insert_into2.q, \
   insert_into3.q, \
@@ -1048,6 +1051,7 @@ spark.query.files=add_part_multiple.q, \
   join38.q, \
   join39.q, \
   join4.q, \
+  join40.q, \
   join41.q, \
   join5.q, \
   join6.q, \
@@ -1066,6 +1070,7 @@ spark.query.files=add_part_multiple.q, \
   join_cond_pushdown_unqual2.q, \
   join_cond_pushdown_unqual3.q, \
   join_cond_pushdown_unqual4.q, \
+  join_empty.q, \
   join_filters_overlap.q, \
   join_hive_626.q, \
   join_literals.q, \
@@ -1151,6 +1156,7 @@ spark.query.files=add_part_multiple.q, \
   ppd_join.q, \
   ppd_join2.q, \
   ppd_join3.q, \
+  ppd_join4.q, \
   ppd_join5.q, \
   ppd_join_filter.q, \
   ppd_multi_insert.q, \
@@ -1272,6 +1278,7 @@ spark.query.files=add_part_multiple.q, \
   timestamp_lazy.q, \
   timestamp_null.q, \
   timestamp_udf.q, \
+  transform1.q, \
   transform2.q, \
   transform_ppr1.q, \
   transform_ppr2.q, \

http://git-wip-us.apache.org/repos/asf/hive/blob/2948c160/ql/src/test/results/clientpositive/spark/auto_join32.q.out
--
diff --git a/ql/src/test/results/clientpositive/spark/auto_join32.q.out 
b/ql/src/test/results/clientpositive/spark/auto_join32.q.out
new file mode 100644
index 000..688b88d
--- /dev/null
+++ b/ql/src/test/results/clientpositive/spark/auto_join32.q.out
@@ -0,0 +1,574 @@
+PREHOOK: query: create table studenttab10k (name string, age int, gpa double)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@studenttab10k
+POSTHOOK: query: create table studenttab10k (name string, age int, gpa double)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@studenttab10k
+PREHOOK: query: create table votertab10k (name string, age int, registration 
string, contributions float)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@votertab10k
+POSTHOOK: query: create table votertab10k (name string, age int, registration 
string, contributions float)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@votertab10k
+PREHOOK: query: explain select s.name, count(distinct registration)
+from studenttab10k s join votertab10k v
+on (s.name = v.name)
+group by s.name
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select s.name, count(distinct registration)
+from studenttab10k s join votertab10k v
+on (s.name = v.name)
+group by s.name
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+  Stage-2 is a root stage
+  Stage-1 depends on stages: Stage-2
+  Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+  Stage: Stage-2
+Spark
+ A masked patt

[2/3] hive git commit: HIVE-13843: Re-enable the HoS tests disabled in HIVE-13402 (Sahil Takiar, reviewed by Rui Li)

2017-10-01 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/2948c160/ql/src/test/results/clientpositive/spark/join40.q.out
--
diff --git a/ql/src/test/results/clientpositive/spark/join40.q.out 
b/ql/src/test/results/clientpositive/spark/join40.q.out
new file mode 100644
index 000..cd40042
--- /dev/null
+++ b/ql/src/test/results/clientpositive/spark/join40.q.out
@@ -0,0 +1,3845 @@
+PREHOOK: query: EXPLAIN SELECT x.key, x.value, y.key, y.value
+FROM src x left outer JOIN (select * from src where key <= 100) y ON (x.key = 
y.key)
+PREHOOK: type: QUERY
+POSTHOOK: query: EXPLAIN SELECT x.key, x.value, y.key, y.value
+FROM src x left outer JOIN (select * from src where key <= 100) y ON (x.key = 
y.key)
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+  Stage-1 is a root stage
+  Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+  Stage: Stage-1
+Spark
+  Edges:
+Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Map 3 (PARTITION-LEVEL 
SORT, 2)
+ A masked pattern was here 
+  Vertices:
+Map 1 
+Map Operator Tree:
+TableScan
+  alias: x
+  Statistics: Num rows: 500 Data size: 5312 Basic stats: 
COMPLETE Column stats: NONE
+  Select Operator
+expressions: key (type: string), value (type: string)
+outputColumnNames: _col0, _col1
+Statistics: Num rows: 500 Data size: 5312 Basic stats: 
COMPLETE Column stats: NONE
+Reduce Output Operator
+  key expressions: _col0 (type: string)
+  sort order: +
+  Map-reduce partition columns: _col0 (type: string)
+  Statistics: Num rows: 500 Data size: 5312 Basic stats: 
COMPLETE Column stats: NONE
+  value expressions: _col1 (type: string)
+Map 3 
+Map Operator Tree:
+TableScan
+  alias: src
+  Statistics: Num rows: 500 Data size: 5312 Basic stats: 
COMPLETE Column stats: NONE
+  Filter Operator
+predicate: (UDFToDouble(key) <= 100.0) (type: boolean)
+Statistics: Num rows: 166 Data size: 1763 Basic stats: 
COMPLETE Column stats: NONE
+Select Operator
+  expressions: key (type: string), value (type: string)
+  outputColumnNames: _col0, _col1
+  Statistics: Num rows: 166 Data size: 1763 Basic stats: 
COMPLETE Column stats: NONE
+  Reduce Output Operator
+key expressions: _col0 (type: string)
+sort order: +
+Map-reduce partition columns: _col0 (type: string)
+Statistics: Num rows: 166 Data size: 1763 Basic stats: 
COMPLETE Column stats: NONE
+value expressions: _col1 (type: string)
+Reducer 2 
+Reduce Operator Tree:
+  Join Operator
+condition map:
+ Left Outer Join 0 to 1
+keys:
+  0 _col0 (type: string)
+  1 _col0 (type: string)
+outputColumnNames: _col0, _col1, _col2, _col3
+Statistics: Num rows: 550 Data size: 5843 Basic stats: 
COMPLETE Column stats: NONE
+File Output Operator
+  compressed: false
+  Statistics: Num rows: 550 Data size: 5843 Basic stats: 
COMPLETE Column stats: NONE
+  table:
+  input format: 
org.apache.hadoop.mapred.SequenceFileInputFormat
+  output format: 
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+  Stage: Stage-0
+Fetch Operator
+  limit: -1
+  Processor Tree:
+ListSink
+
+PREHOOK: query: SELECT x.key, x.value, y.key, y.value
+FROM src x left outer JOIN (select * from src where key <= 100) y ON (x.key = 
y.key)
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src
+ A masked pattern was here 
+POSTHOOK: query: SELECT x.key, x.value, y.key, y.value
+FROM src x left outer JOIN (select * from src where key <= 100) y ON (x.key = 
y.key)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src
+ A masked pattern was here 
+0  val_0   0   val_0
+0  val_0   0   val_0
+0  val_0   0   val_0
+0  val_0   0   val_0
+0  val_0   0   val_0
+0  val_0   0   val_0
+0  val_0   0   val_0
+0  val_0   0   val_0
+0  val_0   0   val_0
+10 val_10  10  val_10
+100val_100 100 val_100
+100val_100 100 val_100
+100val_100 100 val_100
+100val_100 100 val_100
+103val_103 NULLNULL
+103val_103 NULLNULL
+104val_1

[1/3] hive git commit: HIVE-13843: Re-enable the HoS tests disabled in HIVE-13402 (Sahil Takiar, reviewed by Rui Li)

2017-10-01 Thread stakiar

Repository: hive
Updated Branches:
  refs/heads/master 9244fdc7e -> 2948c160f


http://git-wip-us.apache.org/repos/asf/hive/blob/2948c160/ql/src/test/results/clientpositive/spark/join_empty.q.out
--
diff --git a/ql/src/test/results/clientpositive/spark/join_empty.q.out 
b/ql/src/test/results/clientpositive/spark/join_empty.q.out
new file mode 100644
index 000..344345b
--- /dev/null
+++ b/ql/src/test/results/clientpositive/spark/join_empty.q.out
@@ -0,0 +1,56 @@
+PREHOOK: query: create table srcpart_empty(key int, value string) partitioned 
by (ds string)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@srcpart_empty
+POSTHOOK: query: create table srcpart_empty(key int, value string) partitioned 
by (ds string)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@srcpart_empty
+PREHOOK: query: create table src2_empty (key int, value string)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@src2_empty
+POSTHOOK: query: create table src2_empty (key int, value string)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@src2_empty
+PREHOOK: query: select /*+mapjoin(a)*/ a.key, b.value from srcpart_empty a 
join src b on a.key=b.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src
+PREHOOK: Input: default@srcpart_empty
+ A masked pattern was here 
+POSTHOOK: query: select /*+mapjoin(a)*/ a.key, b.value from srcpart_empty a 
join src b on a.key=b.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src
+POSTHOOK: Input: default@srcpart_empty
+ A masked pattern was here 
+PREHOOK: query: select /*+mapjoin(a)*/ a.key, b.value from src2_empty a join 
src b on a.key=b.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src
+PREHOOK: Input: default@src2_empty
+ A masked pattern was here 
+POSTHOOK: query: select /*+mapjoin(a)*/ a.key, b.value from src2_empty a join 
src b on a.key=b.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src
+POSTHOOK: Input: default@src2_empty
+ A masked pattern was here 
+PREHOOK: query: select a.key, b.value from srcpart_empty a join src b on 
a.key=b.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src
+PREHOOK: Input: default@srcpart_empty
+ A masked pattern was here 
+POSTHOOK: query: select a.key, b.value from srcpart_empty a join src b on 
a.key=b.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src
+POSTHOOK: Input: default@srcpart_empty
+ A masked pattern was here 
+PREHOOK: query: select a.key, b.value from src2_empty a join src b on 
a.key=b.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src
+PREHOOK: Input: default@src2_empty
+ A masked pattern was here 
+POSTHOOK: query: select a.key, b.value from src2_empty a join src b on 
a.key=b.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src
+POSTHOOK: Input: default@src2_empty
+ A masked pattern was here 

http://git-wip-us.apache.org/repos/asf/hive/blob/2948c160/ql/src/test/results/clientpositive/spark/ppd_join4.q.out
--
diff --git a/ql/src/test/results/clientpositive/spark/ppd_join4.q.out 
b/ql/src/test/results/clientpositive/spark/ppd_join4.q.out
new file mode 100644
index 000..2e80db1
--- /dev/null
+++ b/ql/src/test/results/clientpositive/spark/ppd_join4.q.out
@@ -0,0 +1,128 @@
+PREHOOK: query: create table dual(a string)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@dual
+POSTHOOK: query: create table dual(a string)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@dual
+PREHOOK: query: drop table if exists test_tbl
+PREHOOK: type: DROPTABLE
+POSTHOOK: query: drop table if exists test_tbl
+POSTHOOK: type: DROPTABLE
+PREHOOK: query: create table test_tbl (id string,name string)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@test_tbl
+POSTHOOK: query: create table test_tbl (id string,name string)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@test_tbl
+PREHOOK: query: insert into table test_tbl
+select 'a','b' from dual
+PREHOOK: type: QUERY
+PREHOOK: Input: default@dual
+PREHOOK: Output: default@test_tbl
+POSTHOOK: query: insert into table test_tbl
+select 'a','b' from dual
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@dual
+POSTHOOK: Output: default@test_tbl
+POSTHOOK: Lineage: test_tbl.id SIMPLE []
+POSTHOOK: Lineage: test_tbl.name SIMPLE []
+PREHOOK: query: explain
+select t2.* 
+from
+(select id,name from (select id,name from test_tbl) t1 sort by id) t2
+join test_tbl t3 on (t2.id=t3.id )
+where t2.name='c' and t3.id='a'
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select t2.* 
+from
+(select id,name from (select id,name from test_tbl) t1 sort by id) t2
+join test_tbl t3 o

[40/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query13.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query13.q.out 
b/ql/src/test/results/clientpositive/perf/query13.q.out
deleted file mode 100644
index 7cb5449..000
--- a/ql/src/test/results/clientpositive/perf/query13.q.out
+++ /dev/null
@@ -1,205 +0,0 @@
-PREHOOK: query: explain
-select avg(ss_quantity)
-   ,avg(ss_ext_sales_price)
-   ,avg(ss_ext_wholesale_cost)
-   ,sum(ss_ext_wholesale_cost)
- from store_sales
- ,store
- ,customer_demographics
- ,household_demographics
- ,customer_address
- ,date_dim
- where s_store_sk = ss_store_sk
- and  ss_sold_date_sk = d_date_sk and d_year = 2001
- and((ss_hdemo_sk=hd_demo_sk
-  and cd_demo_sk = ss_cdemo_sk
-  and cd_marital_status = 'M'
-  and cd_education_status = '4 yr Degree'
-  and ss_sales_price between 100.00 and 150.00
-  and hd_dep_count = 3   
- )or
- (ss_hdemo_sk=hd_demo_sk
-  and cd_demo_sk = ss_cdemo_sk
-  and cd_marital_status = 'D'
-  and cd_education_status = 'Primary'
-  and ss_sales_price between 50.00 and 100.00   
-  and hd_dep_count = 1
- ) or 
- (ss_hdemo_sk=hd_demo_sk
-  and cd_demo_sk = ss_cdemo_sk
-  and cd_marital_status = 'U'
-  and cd_education_status = 'Advanced Degree'
-  and ss_sales_price between 150.00 and 200.00 
-  and hd_dep_count = 1  
- ))
- and((ss_addr_sk = ca_address_sk
-  and ca_country = 'United States'
-  and ca_state in ('KY', 'GA', 'NM')
-  and ss_net_profit between 100 and 200  
- ) or
- (ss_addr_sk = ca_address_sk
-  and ca_country = 'United States'
-  and ca_state in ('MT', 'OR', 'IN')
-  and ss_net_profit between 150 and 300  
- ) or
- (ss_addr_sk = ca_address_sk
-  and ca_country = 'United States'
-  and ca_state in ('WI', 'MO', 'WV')
-  and ss_net_profit between 50 and 250  
- ))
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-select avg(ss_quantity)
-   ,avg(ss_ext_sales_price)
-   ,avg(ss_ext_wholesale_cost)
-   ,sum(ss_ext_wholesale_cost)
- from store_sales
- ,store
- ,customer_demographics
- ,household_demographics
- ,customer_address
- ,date_dim
- where s_store_sk = ss_store_sk
- and  ss_sold_date_sk = d_date_sk and d_year = 2001
- and((ss_hdemo_sk=hd_demo_sk
-  and cd_demo_sk = ss_cdemo_sk
-  and cd_marital_status = 'M'
-  and cd_education_status = '4 yr Degree'
-  and ss_sales_price between 100.00 and 150.00
-  and hd_dep_count = 3   
- )or
- (ss_hdemo_sk=hd_demo_sk
-  and cd_demo_sk = ss_cdemo_sk
-  and cd_marital_status = 'D'
-  and cd_education_status = 'Primary'
-  and ss_sales_price between 50.00 and 100.00   
-  and hd_dep_count = 1
- ) or 
- (ss_hdemo_sk=hd_demo_sk
-  and cd_demo_sk = ss_cdemo_sk
-  and cd_marital_status = 'U'
-  and cd_education_status = 'Advanced Degree'
-  and ss_sales_price between 150.00 and 200.00 
-  and hd_dep_count = 1  
- ))
- and((ss_addr_sk = ca_address_sk
-  and ca_country = 'United States'
-  and ca_state in ('KY', 'GA', 'NM')
-  and ss_net_profit between 100 and 200  
- ) or
- (ss_addr_sk = ca_address_sk
-  and ca_country = 'United States'
-  and ca_state in ('MT', 'OR', 'IN')
-  and ss_net_profit between 150 and 300  
- ) or
- (ss_addr_sk = ca_address_sk
-  and ca_country = 'United States'
-  and ca_state in ('WI', 'MO', 'WV')
-  and ss_net_profit between 50 and 250  
- ))
-POSTHOOK: type: QUERY
-Plan optimized by CBO.
-
-Vertex dependency in root stage
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 8 (SIMPLE_EDGE)
-Reducer 3 <- Map 9 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Map 10 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
-Reducer 5 <- Map 11 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
-Reducer 6 <- Map 12 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
-Reducer 7 <- Reducer 6 (CUSTOM_SIMPLE_EDGE)
-
-Stage-0
-  Fetch Operator
-limit:-1
-Stage-1
-  Reducer 7
-  File Output Operator [FS_40]
-Group By Operator [GBY_38] (rows=1 width=764)
-  
Output:["_col0","_col1","_col2","_col3"],aggregations:["avg(VALUE._col0)","avg(VALUE._col1)","avg(VALUE._col2)","sum(VALUE._col3)"]
-<-Reducer 6 [CUSTOM_SIMPLE_EDGE]
-  PARTITION_ONLY_SHUFFLE [RS_37]
-Group By Operator [GBY_36] (rows=1 width=764)
-  
Output:["_col0","_col1","_col2","_col3"],aggregations:["avg(_col6)","avg(_col8)","avg(_col9)","sum(_col9)"]
-  Select Operator [SEL_35] (rows=715776 width=88)
-Output:["_col6","_col8","_col9"]
-Filter Operator [FIL_34] (rows=715776 width=88)
-  predicate:(((_col19 = 'D') and (_col20 = 'Primary') and 
_col7 BETWEEN 50 AND 100 and (_col14 = 1)) or ((_col19 = 'M') and (_col20 = '4 
yr Degree') and _col7 BETWEEN 100 AND 150 and (_col14 = 3)) or ((_col19 = 'U') 
and (_col20 = 'Advanced Degree') and _col7 BETWEEN 150 AND 200 and (_col14 = 
1)))

[06/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query75.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query75.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query75.q.out
new file mode 100644
index 000..08e1620
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query75.q.out
@@ -0,0 +1,481 @@
+PREHOOK: query: explain
+WITH all_sales AS (
+ SELECT d_year
+   ,i_brand_id
+   ,i_class_id
+   ,i_category_id
+   ,i_manufact_id
+   ,SUM(sales_cnt) AS sales_cnt
+   ,SUM(sales_amt) AS sales_amt
+ FROM (SELECT d_year
+ ,i_brand_id
+ ,i_class_id
+ ,i_category_id
+ ,i_manufact_id
+ ,cs_quantity - COALESCE(cr_return_quantity,0) AS sales_cnt
+ ,cs_ext_sales_price - COALESCE(cr_return_amount,0.0) AS sales_amt
+   FROM catalog_sales JOIN item ON i_item_sk=cs_item_sk
+  JOIN date_dim ON d_date_sk=cs_sold_date_sk
+  LEFT JOIN catalog_returns ON 
(cs_order_number=cr_order_number 
+AND cs_item_sk=cr_item_sk)
+   WHERE i_category='Sports'
+   UNION
+   SELECT d_year
+ ,i_brand_id
+ ,i_class_id
+ ,i_category_id
+ ,i_manufact_id
+ ,ss_quantity - COALESCE(sr_return_quantity,0) AS sales_cnt
+ ,ss_ext_sales_price - COALESCE(sr_return_amt,0.0) AS sales_amt
+   FROM store_sales JOIN item ON i_item_sk=ss_item_sk
+JOIN date_dim ON d_date_sk=ss_sold_date_sk
+LEFT JOIN store_returns ON 
(ss_ticket_number=sr_ticket_number 
+AND ss_item_sk=sr_item_sk)
+   WHERE i_category='Sports'
+   UNION
+   SELECT d_year
+ ,i_brand_id
+ ,i_class_id
+ ,i_category_id
+ ,i_manufact_id
+ ,ws_quantity - COALESCE(wr_return_quantity,0) AS sales_cnt
+ ,ws_ext_sales_price - COALESCE(wr_return_amt,0.0) AS sales_amt
+   FROM web_sales JOIN item ON i_item_sk=ws_item_sk
+  JOIN date_dim ON d_date_sk=ws_sold_date_sk
+  LEFT JOIN web_returns ON 
(ws_order_number=wr_order_number 
+AND ws_item_sk=wr_item_sk)
+   WHERE i_category='Sports') sales_detail
+ GROUP BY d_year, i_brand_id, i_class_id, i_category_id, i_manufact_id)
+ SELECT  prev_yr.d_year AS prev_year
+  ,curr_yr.d_year AS year
+  ,curr_yr.i_brand_id
+  ,curr_yr.i_class_id
+  ,curr_yr.i_category_id
+  ,curr_yr.i_manufact_id
+  ,prev_yr.sales_cnt AS prev_yr_cnt
+  ,curr_yr.sales_cnt AS curr_yr_cnt
+  ,curr_yr.sales_cnt-prev_yr.sales_cnt AS 
sales_cnt_diff
+  ,curr_yr.sales_amt-prev_yr.sales_amt AS 
sales_amt_diff
+ FROM all_sales curr_yr, all_sales prev_yr
+ WHERE curr_yr.i_brand_id=prev_yr.i_brand_id
+   AND curr_yr.i_class_id=prev_yr.i_class_id
+   AND curr_yr.i_category_id=prev_yr.i_category_id
+   AND curr_yr.i_manufact_id=prev_yr.i_manufact_id
+   AND curr_yr.d_year=2002
+   AND prev_yr.d_year=2002-1
+   AND CAST(curr_yr.sales_cnt AS DECIMAL(17,2))/CAST(prev_yr.sales_cnt AS 
DECIMAL(17,2))<0.9
+ ORDER BY sales_cnt_diff
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+WITH all_sales AS (
+ SELECT d_year
+   ,i_brand_id
+   ,i_class_id
+   ,i_category_id
+   ,i_manufact_id
+   ,SUM(sales_cnt) AS sales_cnt
+   ,SUM(sales_amt) AS sales_amt
+ FROM (SELECT d_year
+ ,i_brand_id
+ ,i_class_id
+ ,i_category_id
+ ,i_manufact_id
+ ,cs_quantity - COALESCE(cr_return_quantity,0) AS sales_cnt
+ ,cs_ext_sales_price - COALESCE(cr_return_amount,0.0) AS sales_amt
+   FROM catalog_sales JOIN item ON i_item_sk=cs_item_sk
+  JOIN date_dim ON d_date_sk=cs_sold_date_sk
+  LEFT JOIN catalog_returns ON 
(cs_order_number=cr_order_number 
+AND cs_item_sk=cr_item_sk)
+   WHERE i_category='Sports'
+   UNION
+   SELECT d_year
+ ,i_brand_id
+ ,i_class_id
+ ,i_category_id
+ ,i_manufact_id
+ ,ss_quantity - COALESCE(sr_return_quantity,0) AS sales_cnt
+ ,ss_ext_sales_price - COALESCE(sr_return_amt,0.0) AS sales_amt
+   FROM store_sales JOIN item ON i_item_sk=ss_item_sk
+JOIN date_dim ON d_date_sk=ss_sold_date_sk
+LEFT JOIN store_returns ON 
(ss_ticket_number=sr_ticket_number

[24/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query81.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query81.q.out 
b/ql/src/test/results/clientpositive/perf/query81.q.out
deleted file mode 100644
index 5fb04b2..000
--- a/ql/src/test/results/clientpositive/perf/query81.q.out
+++ /dev/null
@@ -1,214 +0,0 @@
-PREHOOK: query: explain
-with customer_total_return as
- (select cr_returning_customer_sk as ctr_customer_sk
-,ca_state as ctr_state, 
-   sum(cr_return_amt_inc_tax) as ctr_total_return
- from catalog_returns
- ,date_dim
- ,customer_address
- where cr_returned_date_sk = d_date_sk 
-   and d_year =1998
-   and cr_returning_addr_sk = ca_address_sk 
- group by cr_returning_customer_sk
- ,ca_state )
-  select  
c_customer_id,c_salutation,c_first_name,c_last_name,ca_street_number,ca_street_name
-   
,ca_street_type,ca_suite_number,ca_city,ca_county,ca_state,ca_zip,ca_country,ca_gmt_offset
-  ,ca_location_type,ctr_total_return
- from customer_total_return ctr1
- ,customer_address
- ,customer
- where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2
- from customer_total_return ctr2 
- where ctr1.ctr_state = ctr2.ctr_state)
-   and ca_address_sk = c_current_addr_sk
-   and ca_state = 'IL'
-   and ctr1.ctr_customer_sk = c_customer_sk
- order by 
c_customer_id,c_salutation,c_first_name,c_last_name,ca_street_number,ca_street_name
-   
,ca_street_type,ca_suite_number,ca_city,ca_county,ca_state,ca_zip,ca_country,ca_gmt_offset
-  ,ca_location_type,ctr_total_return
- limit 100
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-with customer_total_return as
- (select cr_returning_customer_sk as ctr_customer_sk
-,ca_state as ctr_state, 
-   sum(cr_return_amt_inc_tax) as ctr_total_return
- from catalog_returns
- ,date_dim
- ,customer_address
- where cr_returned_date_sk = d_date_sk 
-   and d_year =1998
-   and cr_returning_addr_sk = ca_address_sk 
- group by cr_returning_customer_sk
- ,ca_state )
-  select  
c_customer_id,c_salutation,c_first_name,c_last_name,ca_street_number,ca_street_name
-   
,ca_street_type,ca_suite_number,ca_city,ca_county,ca_state,ca_zip,ca_country,ca_gmt_offset
-  ,ca_location_type,ctr_total_return
- from customer_total_return ctr1
- ,customer_address
- ,customer
- where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2
- from customer_total_return ctr2 
- where ctr1.ctr_state = ctr2.ctr_state)
-   and ca_address_sk = c_current_addr_sk
-   and ca_state = 'IL'
-   and ctr1.ctr_customer_sk = c_customer_sk
- order by 
c_customer_id,c_salutation,c_first_name,c_last_name,ca_street_number,ca_street_name
-   
,ca_street_type,ca_suite_number,ca_city,ca_county,ca_state,ca_zip,ca_country,ca_gmt_offset
-  ,ca_location_type,ctr_total_return
- limit 100
-POSTHOOK: type: QUERY
-Plan optimized by CBO.
-
-Vertex dependency in root stage
-Reducer 10 <- Reducer 13 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
-Reducer 11 <- Map 14 (SIMPLE_EDGE), Map 6 (SIMPLE_EDGE)
-Reducer 12 <- Map 15 (SIMPLE_EDGE), Reducer 11 (SIMPLE_EDGE)
-Reducer 13 <- Reducer 12 (SIMPLE_EDGE)
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 5 (SIMPLE_EDGE)
-Reducer 3 <- Reducer 10 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
-Reducer 7 <- Map 14 (SIMPLE_EDGE), Map 6 (SIMPLE_EDGE)
-Reducer 8 <- Map 15 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
-Reducer 9 <- Reducer 8 (SIMPLE_EDGE)
-
-Stage-0
-  Fetch Operator
-limit:-1
-Stage-1
-  Reducer 4
-  File Output Operator [FS_67]
-Select Operator [SEL_66] (rows=100 width=860)
-  
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13","_col14","_col15"]
-  Limit [LIM_65] (rows=100 width=860)
-Number of rows:100
-Select Operator [SEL_64] (rows=9683 width=860)
-  
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13","_col14"]
-<-Reducer 3 [SIMPLE_EDGE]
-  SHUFFLE [RS_63]
-Select Operator [SEL_62] (rows=9683 width=860)
-  
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13","_col14"]
-  Merge Join Operator [MERGEJOIN_108] (rows=9683 width=860)
-
Conds:RS_59._col0=RS_60._col0(Inner),Output:["_col1","_col3","_col4","_col5","_col7","_col8","_col9","_col10","_col11","_col12","_col14","_col15","_col16","_col17","_col20"]
-  <-Reducer

[25/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query78.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query78.q.out 
b/ql/src/test/results/clientpositive/perf/query78.q.out
deleted file mode 100644
index 91c244f..000
--- a/ql/src/test/results/clientpositive/perf/query78.q.out
+++ /dev/null
@@ -1,292 +0,0 @@
-PREHOOK: query: explain
-with ws as
-  (select d_year AS ws_sold_year, ws_item_sk,
-ws_bill_customer_sk ws_customer_sk,
-sum(ws_quantity) ws_qty,
-sum(ws_wholesale_cost) ws_wc,
-sum(ws_sales_price) ws_sp
-   from web_sales
-   left join web_returns on wr_order_number=ws_order_number and 
ws_item_sk=wr_item_sk
-   join date_dim on ws_sold_date_sk = d_date_sk
-   where wr_order_number is null
-   group by d_year, ws_item_sk, ws_bill_customer_sk
-   ),
-cs as
-  (select d_year AS cs_sold_year, cs_item_sk,
-cs_bill_customer_sk cs_customer_sk,
-sum(cs_quantity) cs_qty,
-sum(cs_wholesale_cost) cs_wc,
-sum(cs_sales_price) cs_sp
-   from catalog_sales
-   left join catalog_returns on cr_order_number=cs_order_number and 
cs_item_sk=cr_item_sk
-   join date_dim on cs_sold_date_sk = d_date_sk
-   where cr_order_number is null
-   group by d_year, cs_item_sk, cs_bill_customer_sk
-   ),
-ss as
-  (select d_year AS ss_sold_year, ss_item_sk,
-ss_customer_sk,
-sum(ss_quantity) ss_qty,
-sum(ss_wholesale_cost) ss_wc,
-sum(ss_sales_price) ss_sp
-   from store_sales
-   left join store_returns on sr_ticket_number=ss_ticket_number and 
ss_item_sk=sr_item_sk
-   join date_dim on ss_sold_date_sk = d_date_sk
-   where sr_ticket_number is null
-   group by d_year, ss_item_sk, ss_customer_sk
-   )
- select 
-ss_sold_year, ss_item_sk, ss_customer_sk,
-round(ss_qty/(coalesce(ws_qty+cs_qty,1)),2) ratio,
-ss_qty store_qty, ss_wc store_wholesale_cost, ss_sp store_sales_price,
-coalesce(ws_qty,0)+coalesce(cs_qty,0) other_chan_qty,
-coalesce(ws_wc,0)+coalesce(cs_wc,0) other_chan_wholesale_cost,
-coalesce(ws_sp,0)+coalesce(cs_sp,0) other_chan_sales_price
-from ss
-left join ws on (ws_sold_year=ss_sold_year and ws_item_sk=ss_item_sk and 
ws_customer_sk=ss_customer_sk)
-left join cs on (cs_sold_year=ss_sold_year and cs_item_sk=cs_item_sk and 
cs_customer_sk=ss_customer_sk)
-where coalesce(ws_qty,0)>0 and coalesce(cs_qty, 0)>0 and ss_sold_year=2000
-order by 
-  ss_sold_year, ss_item_sk, ss_customer_sk,
-  ss_qty desc, ss_wc desc, ss_sp desc,
-  other_chan_qty,
-  other_chan_wholesale_cost,
-  other_chan_sales_price,
-  round(ss_qty/(coalesce(ws_qty+cs_qty,1)),2)
-limit 100
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-with ws as
-  (select d_year AS ws_sold_year, ws_item_sk,
-ws_bill_customer_sk ws_customer_sk,
-sum(ws_quantity) ws_qty,
-sum(ws_wholesale_cost) ws_wc,
-sum(ws_sales_price) ws_sp
-   from web_sales
-   left join web_returns on wr_order_number=ws_order_number and 
ws_item_sk=wr_item_sk
-   join date_dim on ws_sold_date_sk = d_date_sk
-   where wr_order_number is null
-   group by d_year, ws_item_sk, ws_bill_customer_sk
-   ),
-cs as
-  (select d_year AS cs_sold_year, cs_item_sk,
-cs_bill_customer_sk cs_customer_sk,
-sum(cs_quantity) cs_qty,
-sum(cs_wholesale_cost) cs_wc,
-sum(cs_sales_price) cs_sp
-   from catalog_sales
-   left join catalog_returns on cr_order_number=cs_order_number and 
cs_item_sk=cr_item_sk
-   join date_dim on cs_sold_date_sk = d_date_sk
-   where cr_order_number is null
-   group by d_year, cs_item_sk, cs_bill_customer_sk
-   ),
-ss as
-  (select d_year AS ss_sold_year, ss_item_sk,
-ss_customer_sk,
-sum(ss_quantity) ss_qty,
-sum(ss_wholesale_cost) ss_wc,
-sum(ss_sales_price) ss_sp
-   from store_sales
-   left join store_returns on sr_ticket_number=ss_ticket_number and 
ss_item_sk=sr_item_sk
-   join date_dim on ss_sold_date_sk = d_date_sk
-   where sr_ticket_number is null
-   group by d_year, ss_item_sk, ss_customer_sk
-   )
- select 
-ss_sold_year, ss_item_sk, ss_customer_sk,
-round(ss_qty/(coalesce(ws_qty+cs_qty,1)),2) ratio,
-ss_qty store_qty, ss_wc store_wholesale_cost, ss_sp store_sales_price,
-coalesce(ws_qty,0)+coalesce(cs_qty,0) other_chan_qty,
-coalesce(ws_wc,0)+coalesce(cs_wc,0) other_chan_wholesale_cost,
-coalesce(ws_sp,0)+coalesce(cs_sp,0) other_chan_sales_price
-from ss
-left join ws on (ws_sold_year=ss_sold_year and ws_item_sk=ss_item_sk and 
ws_customer_sk=ss_customer_sk)
-left join cs on (cs_sold_year=ss_sold_year and cs_item_sk=cs_item_sk and 
cs_customer_sk=ss_customer_sk)
-where coalesce(ws_qty,0)>0 and coalesce(cs_qty, 0)>0 and ss_sold_year=2000
-order by 
-  ss_sold_year, ss_item_sk, ss_customer_sk,
-  ss_qty desc, ss_wc desc, ss_sp desc,
-  other_chan_qty,
-  other_chan_wholesale_cost,
-  other_chan_sales_price,
-  round(ss_qty/(coalesce(ws_qty+cs_qty,1)),2)
-limit 100
-POSTHOOK: type: QUERY
-Plan optimized by CBO.
-
-Vertex dependency in root sta

[17/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query22.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query22.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query22.q.out
new file mode 100644
index 000..995f61d
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query22.q.out
@@ -0,0 +1,121 @@
+PREHOOK: query: explain
+select  i_product_name
+ ,i_brand
+ ,i_class
+ ,i_category
+ ,avg(inv_quantity_on_hand) qoh
+   from inventory
+   ,date_dim
+   ,item
+   ,warehouse
+   where inv_date_sk=d_date_sk
+  and inv_item_sk=i_item_sk
+  and inv_warehouse_sk = w_warehouse_sk
+  and d_month_seq between 1212 and 1212 + 11
+   group by rollup(i_product_name
+   ,i_brand
+   ,i_class
+   ,i_category)
+order by qoh, i_product_name, i_brand, i_class, i_category
+limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select  i_product_name
+ ,i_brand
+ ,i_class
+ ,i_category
+ ,avg(inv_quantity_on_hand) qoh
+   from inventory
+   ,date_dim
+   ,item
+   ,warehouse
+   where inv_date_sk=d_date_sk
+  and inv_item_sk=i_item_sk
+  and inv_warehouse_sk = w_warehouse_sk
+  and d_month_seq between 1212 and 1212 + 11
+   group by rollup(i_product_name
+   ,i_brand
+   ,i_class
+   ,i_category)
+order by qoh, i_product_name, i_brand, i_class, i_category
+limit 100
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 7 (SIMPLE_EDGE)
+Reducer 3 <- Map 8 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Map 9 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
+Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
+Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
+
+Stage-0
+  Fetch Operator
+limit:100
+Stage-1
+  Reducer 6
+  File Output Operator [FS_30]
+Limit [LIM_29] (rows=100 width=15)
+  Number of rows:100
+  Select Operator [SEL_28] (rows=125060762 width=15)
+Output:["_col0","_col1","_col2","_col3","_col4"]
+  <-Reducer 5 [SIMPLE_EDGE]
+SHUFFLE [RS_27]
+  Select Operator [SEL_25] (rows=125060762 width=15)
+Output:["_col0","_col1","_col2","_col3","_col4"]
+Group By Operator [GBY_24] (rows=125060762 width=15)
+  
Output:["_col0","_col1","_col2","_col3","_col5"],aggregations:["avg(VALUE._col0)"],keys:KEY._col0,
 KEY._col1, KEY._col2, KEY._col3, KEY._col4
+<-Reducer 4 [SIMPLE_EDGE]
+  SHUFFLE [RS_23]
+PartitionCols:_col0, _col1, _col2, _col3, _col4
+Group By Operator [GBY_22] (rows=250121525 width=15)
+  
Output:["_col0","_col1","_col2","_col3","_col4","_col5"],aggregations:["avg(_col3)"],keys:_col8,
 _col9, _col10, _col11, 0
+  Merge Join Operator [MERGEJOIN_46] (rows=50024305 
width=15)
+
Conds:RS_18._col1=RS_19._col0(Inner),Output:["_col3","_col8","_col9","_col10","_col11"]
+  <-Map 9 [SIMPLE_EDGE]
+SHUFFLE [RS_19]
+  PartitionCols:_col0
+  Select Operator [SEL_11] (rows=462000 width=1436)
+Output:["_col0","_col1","_col2","_col3","_col4"]
+Filter Operator [FIL_43] (rows=462000 width=1436)
+  predicate:i_item_sk is not null
+  TableScan [TS_9] (rows=462000 width=1436)
+
default@item,item,Tbl:COMPLETE,Col:NONE,Output:["i_item_sk","i_brand","i_class","i_category","i_product_name"]
+  <-Reducer 3 [SIMPLE_EDGE]
+SHUFFLE [RS_18]
+  PartitionCols:_col1
+  Merge Join Operator [MERGEJOIN_45] (rows=45476640 
width=15)
+
Conds:RS_15._col2=RS_16._col0(Inner),Output:["_col1","_col3"]
+  <-Map 8 [SIMPLE_EDGE]
+SHUFFLE [RS_16]
+  PartitionCols:_col0
+  Select Operator [SEL_8] (rows=27 width=1029)
+Output:["_col0"]
+Filter Operator [FIL_42] (rows=27 width=1029)
+  predicate:w_warehouse_sk is not null
+  TableScan [TS_6] (rows=27 width=1029)
+
default@warehouse,warehouse,Tbl:COMPLETE,Col:NONE,Output:["w_warehouse_sk"]
+

[32/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query5.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query5.q.out 
b/ql/src/test/results/clientpositive/perf/query5.q.out
deleted file mode 100644
index ff197d2..000
--- a/ql/src/test/results/clientpositive/perf/query5.q.out
+++ /dev/null
@@ -1,473 +0,0 @@
-PREHOOK: query: explain
-with ssr as
- (select s_store_id,
-sum(sales_price) as sales,
-sum(profit) as profit,
-sum(return_amt) as returns,
-sum(net_loss) as profit_loss
- from
-  ( select  ss_store_sk as store_sk,
-ss_sold_date_sk  as date_sk,
-ss_ext_sales_price as sales_price,
-ss_net_profit as profit,
-cast(0 as decimal(7,2)) as return_amt,
-cast(0 as decimal(7,2)) as net_loss
-from store_sales
-union all
-select sr_store_sk as store_sk,
-   sr_returned_date_sk as date_sk,
-   cast(0 as decimal(7,2)) as sales_price,
-   cast(0 as decimal(7,2)) as profit,
-   sr_return_amt as return_amt,
-   sr_net_loss as net_loss
-from store_returns
-   ) salesreturns,
- date_dim,
- store
- where date_sk = d_date_sk
-   and d_date between cast('1998-08-04' as date) 
-  and (cast('1998-08-04' as date) +  14 days)
-   and store_sk = s_store_sk
- group by s_store_id)
- ,
- csr as
- (select cp_catalog_page_id,
-sum(sales_price) as sales,
-sum(profit) as profit,
-sum(return_amt) as returns,
-sum(net_loss) as profit_loss
- from
-  ( select  cs_catalog_page_sk as page_sk,
-cs_sold_date_sk  as date_sk,
-cs_ext_sales_price as sales_price,
-cs_net_profit as profit,
-cast(0 as decimal(7,2)) as return_amt,
-cast(0 as decimal(7,2)) as net_loss
-from catalog_sales
-union all
-select cr_catalog_page_sk as page_sk,
-   cr_returned_date_sk as date_sk,
-   cast(0 as decimal(7,2)) as sales_price,
-   cast(0 as decimal(7,2)) as profit,
-   cr_return_amount as return_amt,
-   cr_net_loss as net_loss
-from catalog_returns
-   ) salesreturns,
- date_dim,
- catalog_page
- where date_sk = d_date_sk
-   and d_date between cast('1998-08-04' as date)
-  and (cast('1998-08-04' as date) +  14 days)
-   and page_sk = cp_catalog_page_sk
- group by cp_catalog_page_id)
- ,
- wsr as
- (select web_site_id,
-sum(sales_price) as sales,
-sum(profit) as profit,
-sum(return_amt) as returns,
-sum(net_loss) as profit_loss
- from
-  ( select  ws_web_site_sk as wsr_web_site_sk,
-ws_sold_date_sk  as date_sk,
-ws_ext_sales_price as sales_price,
-ws_net_profit as profit,
-cast(0 as decimal(7,2)) as return_amt,
-cast(0 as decimal(7,2)) as net_loss
-from web_sales
-union all
-select ws_web_site_sk as wsr_web_site_sk,
-   wr_returned_date_sk as date_sk,
-   cast(0 as decimal(7,2)) as sales_price,
-   cast(0 as decimal(7,2)) as profit,
-   wr_return_amt as return_amt,
-   wr_net_loss as net_loss
-from web_returns left outer join web_sales on
- ( wr_item_sk = ws_item_sk
-   and wr_order_number = ws_order_number)
-   ) salesreturns,
- date_dim,
- web_site
- where date_sk = d_date_sk
-   and d_date between cast('1998-08-04' as date)
-  and (cast('1998-08-04' as date) +  14 days)
-   and wsr_web_site_sk = web_site_sk
- group by web_site_id)
-  select  channel
-, id
-, sum(sales) as sales
-, sum(returns) as returns
-, sum(profit) as profit
- from 
- (select 'store channel' as channel
-, 'store' || s_store_id as id
-, sales
-, returns
-, (profit - profit_loss) as profit
- from   ssr
- union all
- select 'catalog channel' as channel
-, 'catalog_page' || cp_catalog_page_id as id
-, sales
-, returns
-, (profit - profit_loss) as profit
- from  csr
- union all
- select 'web channel' as channel
-, 'web_site' || web_site_id as id
-, sales
-, returns
-, (profit - profit_loss) as profit
- from   wsr
- ) x
- group by rollup (channel, id)
- order by channel
- ,id
- limit 100
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-with ssr as
- (select s_store_id,
-sum(sales_price) as sales,
-sum(profit) as profit,
-sum(return_amt) as returns,
-sum(net_loss) as profit_loss
- from
-  ( select  ss_store_sk as store_sk,
-ss_sold_date_sk  as date_sk,
-ss_ext_sales_price as sales_price,
-ss_net_profit as profit,
-cast(0 as decimal(7,2)) as return_amt,
-cast(0 as decimal(7,2)) as net_loss
-f

[33/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query45.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query45.q.out 
b/ql/src/test/results/clientpositive/perf/query45.q.out
deleted file mode 100644
index 3efed2e..000
--- a/ql/src/test/results/clientpositive/perf/query45.q.out
+++ /dev/null
@@ -1,180 +0,0 @@
-Warning: Shuffle Join MERGEJOIN[87][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3]] in Stage 'Reducer 4' is a cross product
-PREHOOK: query: explain
-select  ca_zip, ca_county, sum(ws_sales_price)
- from web_sales, customer, customer_address, date_dim, item
- where ws_bill_customer_sk = c_customer_sk
-   and c_current_addr_sk = ca_address_sk 
-   and ws_item_sk = i_item_sk 
-   and ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475', 
'85392', '85460', '80348', '81792')
- or 
- i_item_id in (select i_item_id
- from item
- where i_item_sk in (2, 3, 5, 7, 11, 13, 17, 19, 
23, 29)
- )
-   )
-   and ws_sold_date_sk = d_date_sk
-   and d_qoy = 2 and d_year = 2000
- group by ca_zip, ca_county
- order by ca_zip, ca_county
- limit 100
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-select  ca_zip, ca_county, sum(ws_sales_price)
- from web_sales, customer, customer_address, date_dim, item
- where ws_bill_customer_sk = c_customer_sk
-   and c_current_addr_sk = ca_address_sk 
-   and ws_item_sk = i_item_sk 
-   and ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475', 
'85392', '85460', '80348', '81792')
- or 
- i_item_id in (select i_item_id
- from item
- where i_item_sk in (2, 3, 5, 7, 11, 13, 17, 19, 
23, 29)
- )
-   )
-   and ws_sold_date_sk = d_date_sk
-   and d_qoy = 2 and d_year = 2000
- group by ca_zip, ca_county
- order by ca_zip, ca_county
- limit 100
-POSTHOOK: type: QUERY
-Plan optimized by CBO.
-
-Vertex dependency in root stage
-Reducer 10 <- Reducer 14 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
-Reducer 11 <- Map 8 (SIMPLE_EDGE)
-Reducer 12 <- Map 8 (CUSTOM_SIMPLE_EDGE)
-Reducer 14 <- Map 13 (SIMPLE_EDGE), Map 15 (SIMPLE_EDGE)
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 7 (SIMPLE_EDGE)
-Reducer 3 <- Reducer 10 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Reducer 12 (CUSTOM_SIMPLE_EDGE), Reducer 3 (CUSTOM_SIMPLE_EDGE)
-Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
-Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
-Reducer 9 <- Map 8 (SIMPLE_EDGE), Reducer 11 (ONE_TO_ONE_EDGE)
-
-Stage-0
-  Fetch Operator
-limit:100
-Stage-1
-  Reducer 6
-  File Output Operator [FS_59]
-Limit [LIM_58] (rows=100 width=152)
-  Number of rows:100
-  Select Operator [SEL_57] (rows=95833781 width=152)
-Output:["_col0","_col1","_col2"]
-  <-Reducer 5 [SIMPLE_EDGE]
-SHUFFLE [RS_56]
-  Group By Operator [GBY_54] (rows=95833781 width=152)
-
Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0,
 KEY._col1
-  <-Reducer 4 [SIMPLE_EDGE]
-SHUFFLE [RS_53]
-  PartitionCols:_col0, _col1
-  Group By Operator [GBY_52] (rows=191667562 width=152)
-
Output:["_col0","_col1","_col2"],aggregations:["sum(_col3)"],keys:_col8, _col7
-Select Operator [SEL_51] (rows=191667562 width=152)
-  Output:["_col3","_col7","_col8"]
-  Filter Operator [FIL_50] (rows=191667562 width=152)
-predicate:((substr(_col8, 1, 5)) IN ('85669', '86197', 
'88274', '83405', '86475', '85392', '85460', '80348', '81792') or CASE WHEN 
((_col14 = 0)) THEN (false) WHEN (_col17 is not null) THEN (true) WHEN (_col13 
is null) THEN (null) WHEN ((_col15 < _col14)) THEN (null) ELSE (false) END)
-Select Operator [SEL_49] (rows=191667562 width=152)
-  
Output:["_col3","_col7","_col8","_col13","_col14","_col15","_col17"]
-  Merge Join Operator [MERGEJOIN_87] (rows=191667562 
width=152)
-
Conds:(Inner),Output:["_col3","_col4","_col6","_col8","_col12","_col16","_col17"]
-  <-Reducer 12 [CUSTOM_SIMPLE_EDGE]
-PARTITION_ONLY_SHUFFLE [RS_47]
-  Group By Operator [GBY_38] (rows=1 width=16)
-
Output:["_col0","_col1"],aggregations:["count(VALUE._col0)","count(VALUE._col1)"]
-  <-Map 8 [CUSTOM_SIMPLE_EDGE]
-SHUFFLE [RS_37]
-  Group By Operator [GBY_36] (rows=1 width=16)
-

[22/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query9.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query9.q.out 
b/ql/src/test/results/clientpositive/perf/query9.q.out
deleted file mode 100644
index 24de6f2..000
--- a/ql/src/test/results/clientpositive/perf/query9.q.out
+++ /dev/null
@@ -1,414 +0,0 @@
-Warning: Shuffle Join MERGEJOIN[171][tables = [$hdt$_0, $hdt$_1]] in Stage 
'Reducer 2' is a cross product
-Warning: Shuffle Join MERGEJOIN[172][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in 
Stage 'Reducer 3' is a cross product
-Warning: Shuffle Join MERGEJOIN[173][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3]] in Stage 'Reducer 4' is a cross product
-Warning: Shuffle Join MERGEJOIN[174][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4]] in Stage 'Reducer 5' is a cross product
-Warning: Shuffle Join MERGEJOIN[175][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5]] in Stage 'Reducer 6' is a cross product
-Warning: Shuffle Join MERGEJOIN[176][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6]] in Stage 'Reducer 7' is a cross product
-Warning: Shuffle Join MERGEJOIN[177][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7]] in Stage 'Reducer 8' is a cross 
product
-Warning: Shuffle Join MERGEJOIN[178][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8]] in Stage 'Reducer 9' is 
a cross product
-Warning: Shuffle Join MERGEJOIN[179][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8, $hdt$_9]] in Stage 
'Reducer 10' is a cross product
-Warning: Shuffle Join MERGEJOIN[180][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8, $hdt$_9, $hdt$_10]] in 
Stage 'Reducer 11' is a cross product
-Warning: Shuffle Join MERGEJOIN[181][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8, $hdt$_9, $hdt$_10, 
$hdt$_11]] in Stage 'Reducer 12' is a cross product
-Warning: Shuffle Join MERGEJOIN[182][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8, $hdt$_9, $hdt$_10, 
$hdt$_11, $hdt$_12]] in Stage 'Reducer 13' is a cross product
-Warning: Shuffle Join MERGEJOIN[183][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8, $hdt$_9, $hdt$_10, 
$hdt$_11, $hdt$_12, $hdt$_13]] in Stage 'Reducer 14' is a cross product
-Warning: Shuffle Join MERGEJOIN[184][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8, $hdt$_9, $hdt$_10, 
$hdt$_11, $hdt$_12, $hdt$_13, $hdt$_14]] in Stage 'Reducer 15' is a cross 
product
-Warning: Shuffle Join MERGEJOIN[185][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8, $hdt$_9, $hdt$_10, 
$hdt$_11, $hdt$_12, $hdt$_13, $hdt$_14, $hdt$_15]] in Stage 'Reducer 16' is a 
cross product
-PREHOOK: query: explain
-select case when (select count(*) 
-  from store_sales 
-  where ss_quantity between 1 and 20) > 409437
-then (select avg(ss_ext_list_price) 
-  from store_sales 
-  where ss_quantity between 1 and 20) 
-else (select avg(ss_net_paid_inc_tax)
-  from store_sales
-  where ss_quantity between 1 and 20) end bucket1 ,
-   case when (select count(*)
-  from store_sales
-  where ss_quantity between 21 and 40) > 4595804
-then (select avg(ss_ext_list_price)
-  from store_sales
-  where ss_quantity between 21 and 40) 
-else (select avg(ss_net_paid_inc_tax)
-  from store_sales
-  where ss_quantity between 21 and 40) end bucket2,
-   case when (select count(*)
-  from store_sales
-  where ss_quantity between 41 and 60) > 7887297
-then (select avg(ss_ext_list_price)
-  from store_sales
-  where ss_quantity between 41 and 60)
-else (select avg(ss_net_paid_inc_tax)
-  from store_sales
-  where ss_quantity between 41 and 60) end bucket3,
-   case when (select count(*)
-  from store_sales
-  where ss_quantity between 61 and 80) > 10872978
-then (select avg(ss_ext_list_price)
-  from store_sales
-  where ss_quantity between 61 and 80)
-else (select avg(ss_net_paid_inc_tax)
-  from store_sales
-  where ss_quantity between 61 and 80) end bucket4,
-   case when (select count(*)
-  from store_sales
-  where ss_quantity between 81 and 100) > 43571537
-then (select avg(ss_ext_list_price)

[21/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query95.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query95.q.out 
b/ql/src/test/results/clientpositive/perf/query95.q.out
deleted file mode 100644
index dd3e344..000
--- a/ql/src/test/results/clientpositive/perf/query95.q.out
+++ /dev/null
@@ -1,231 +0,0 @@
-PREHOOK: query: explain
-with ws_wh as
-(select ws1.ws_order_number,ws1.ws_warehouse_sk wh1,ws2.ws_warehouse_sk wh2
- from web_sales ws1,web_sales ws2
- where ws1.ws_order_number = ws2.ws_order_number
-   and ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk)
- select  
-   count(distinct ws_order_number) as `order count`
-  ,sum(ws_ext_ship_cost) as `total shipping cost`
-  ,sum(ws_net_profit) as `total net profit`
-from
-   web_sales ws1
-  ,date_dim
-  ,customer_address
-  ,web_site
-where
-d_date between '1999-5-01' and 
-   (cast('1999-5-01' as date) + 60 days)
-and ws1.ws_ship_date_sk = d_date_sk
-and ws1.ws_ship_addr_sk = ca_address_sk
-and ca_state = 'TX'
-and ws1.ws_web_site_sk = web_site_sk
-and web_company_name = 'pri'
-and ws1.ws_order_number in (select ws_order_number
-from ws_wh)
-and ws1.ws_order_number in (select wr_order_number
-from web_returns,ws_wh
-where wr_order_number = ws_wh.ws_order_number)
-order by count(distinct ws_order_number)
-limit 100
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-with ws_wh as
-(select ws1.ws_order_number,ws1.ws_warehouse_sk wh1,ws2.ws_warehouse_sk wh2
- from web_sales ws1,web_sales ws2
- where ws1.ws_order_number = ws2.ws_order_number
-   and ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk)
- select  
-   count(distinct ws_order_number) as `order count`
-  ,sum(ws_ext_ship_cost) as `total shipping cost`
-  ,sum(ws_net_profit) as `total net profit`
-from
-   web_sales ws1
-  ,date_dim
-  ,customer_address
-  ,web_site
-where
-d_date between '1999-5-01' and 
-   (cast('1999-5-01' as date) + 60 days)
-and ws1.ws_ship_date_sk = d_date_sk
-and ws1.ws_ship_addr_sk = ca_address_sk
-and ca_state = 'TX'
-and ws1.ws_web_site_sk = web_site_sk
-and web_company_name = 'pri'
-and ws1.ws_order_number in (select ws_order_number
-from ws_wh)
-and ws1.ws_order_number in (select wr_order_number
-from web_returns,ws_wh
-where wr_order_number = ws_wh.ws_order_number)
-order by count(distinct ws_order_number)
-limit 100
-POSTHOOK: type: QUERY
-Plan optimized by CBO.
-
-Vertex dependency in root stage
-Reducer 13 <- Map 12 (SIMPLE_EDGE), Map 18 (SIMPLE_EDGE)
-Reducer 14 <- Map 19 (SIMPLE_EDGE), Reducer 13 (SIMPLE_EDGE)
-Reducer 15 <- Reducer 14 (SIMPLE_EDGE)
-Reducer 16 <- Map 12 (SIMPLE_EDGE), Map 18 (SIMPLE_EDGE)
-Reducer 17 <- Reducer 16 (SIMPLE_EDGE)
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 9 (SIMPLE_EDGE)
-Reducer 3 <- Map 10 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Map 11 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
-Reducer 5 <- Reducer 15 (ONE_TO_ONE_EDGE), Reducer 17 (ONE_TO_ONE_EDGE), 
Reducer 4 (SIMPLE_EDGE)
-Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
-Reducer 7 <- Reducer 6 (CUSTOM_SIMPLE_EDGE)
-Reducer 8 <- Reducer 7 (SIMPLE_EDGE)
-
-Stage-0
-  Fetch Operator
-limit:-1
-Stage-1
-  Reducer 8
-  File Output Operator [FS_71]
-Limit [LIM_69] (rows=1 width=344)
-  Number of rows:100
-  Select Operator [SEL_68] (rows=1 width=344)
-Output:["_col0","_col1","_col2"]
-  <-Reducer 7 [SIMPLE_EDGE]
-SHUFFLE [RS_67]
-  Select Operator [SEL_66] (rows=1 width=344)
-Output:["_col1","_col2","_col3"]
-Group By Operator [GBY_120] (rows=1 width=344)
-  
Output:["_col0","_col1","_col2"],aggregations:["count(VALUE._col0)","sum(VALUE._col1)","sum(VALUE._col2)"]
-<-Reducer 6 [CUSTOM_SIMPLE_EDGE]
-  PARTITION_ONLY_SHUFFLE [RS_119]
-Group By Operator [GBY_118] (rows=1 width=344)
-  
Output:["_col0","_col1","_col2"],aggregations:["count(_col0)","sum(_col1)","sum(_col2)"]
-  Group By Operator [GBY_117] (rows=421668645 width=135)
-
Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)","sum(VALUE._col1)"],keys:KEY._col0
-  <-Reducer 5 [SIMPLE_EDGE]
-SHUFFLE [RS_116]
-  PartitionCols:_col0
-  Group By Operator [GBY_115] (rows=421668645 
width=135)
-
Output:["_col0","_col2","_col3"],aggregations:["sum(_col4)","sum(_col5)"],keys:_col3
-Merge Join Operator [MERGEJOIN_127] 
(rows=421668645 width=135)
-  
Conds:RS_58._col3=RS_59._col0(Inner),RS_58._col3=RS_6

[19/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query14.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query14.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query14.q.out
new file mode 100644
index 000..dac6e0f
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query14.q.out
@@ -0,0 +1,1286 @@
+Warning: Shuffle Join MERGEJOIN[892][tables = [$hdt$_2, $hdt$_3, $hdt$_1]] in 
Stage 'Reducer 16' is a cross product
+Warning: Shuffle Join MERGEJOIN[890][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in 
Stage 'Reducer 5' is a cross product
+Warning: Shuffle Join MERGEJOIN[891][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in 
Stage 'Reducer 12' is a cross product
+PREHOOK: query: explain
+with  cross_items as
+ (select i_item_sk ss_item_sk
+ from item,
+ (select iss.i_brand_id brand_id
+ ,iss.i_class_id class_id
+ ,iss.i_category_id category_id
+ from store_sales
+ ,item iss
+ ,date_dim d1
+ where ss_item_sk = iss.i_item_sk
+   and ss_sold_date_sk = d1.d_date_sk
+   and d1.d_year between 1999 AND 1999 + 2
+ intersect 
+ select ics.i_brand_id
+ ,ics.i_class_id
+ ,ics.i_category_id
+ from catalog_sales
+ ,item ics
+ ,date_dim d2
+ where cs_item_sk = ics.i_item_sk
+   and cs_sold_date_sk = d2.d_date_sk
+   and d2.d_year between 1999 AND 1999 + 2
+ intersect
+ select iws.i_brand_id
+ ,iws.i_class_id
+ ,iws.i_category_id
+ from web_sales
+ ,item iws
+ ,date_dim d3
+ where ws_item_sk = iws.i_item_sk
+   and ws_sold_date_sk = d3.d_date_sk
+   and d3.d_year between 1999 AND 1999 + 2) x
+ where i_brand_id = brand_id
+  and i_class_id = class_id
+  and i_category_id = category_id
+),
+ avg_sales as
+ (select avg(quantity*list_price) average_sales
+  from (select ss_quantity quantity
+ ,ss_list_price list_price
+   from store_sales
+   ,date_dim
+   where ss_sold_date_sk = d_date_sk
+ and d_year between 1999 and 2001 
+   union all 
+   select cs_quantity quantity 
+ ,cs_list_price list_price
+   from catalog_sales
+   ,date_dim
+   where cs_sold_date_sk = d_date_sk
+ and d_year between 1998 and 1998 + 2 
+   union all
+   select ws_quantity quantity
+ ,ws_list_price list_price
+   from web_sales
+   ,date_dim
+   where ws_sold_date_sk = d_date_sk
+ and d_year between 1998 and 1998 + 2) x)
+  select  channel, i_brand_id,i_class_id,i_category_id,sum(sales), 
sum(number_sales)
+ from(
+   select 'store' channel, i_brand_id,i_class_id
+ ,i_category_id,sum(ss_quantity*ss_list_price) sales
+ , count(*) number_sales
+   from store_sales
+   ,item
+   ,date_dim
+   where ss_item_sk in (select ss_item_sk from cross_items)
+ and ss_item_sk = i_item_sk
+ and ss_sold_date_sk = d_date_sk
+ and d_year = 1998+2 
+ and d_moy = 11
+   group by i_brand_id,i_class_id,i_category_id
+   having sum(ss_quantity*ss_list_price) > (select average_sales from 
avg_sales)
+   union all
+   select 'catalog' channel, i_brand_id,i_class_id,i_category_id, 
sum(cs_quantity*cs_list_price) sales, count(*) number_sales
+   from catalog_sales
+   ,item
+   ,date_dim
+   where cs_item_sk in (select ss_item_sk from cross_items)
+ and cs_item_sk = i_item_sk
+ and cs_sold_date_sk = d_date_sk
+ and d_year = 1998+2 
+ and d_moy = 11
+   group by i_brand_id,i_class_id,i_category_id
+   having sum(cs_quantity*cs_list_price) > (select average_sales from 
avg_sales)
+   union all
+   select 'web' channel, i_brand_id,i_class_id,i_category_id, 
sum(ws_quantity*ws_list_price) sales , count(*) number_sales
+   from web_sales
+   ,item
+   ,date_dim
+   where ws_item_sk in (select ss_item_sk from cross_items)
+ and ws_item_sk = i_item_sk
+ and ws_sold_date_sk = d_date_sk
+ and d_year = 1998+2
+ and d_moy = 11
+   group by i_brand_id,i_class_id,i_category_id
+   having sum(ws_quantity*ws_list_price) > (select average_sales from 
avg_sales)
+ ) y
+ group by rollup (channel, i_brand_id,i_class_id,i_category_id)
+ order by channel,i_brand_id,i_class_id,i_category_id
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with  cross_items as
+ (select i_item_sk ss_item_sk
+ from item,
+ (select iss.i_brand_id brand_id
+ ,iss.i_class_id class_id
+ ,iss.i_category_id category_id
+ from store_sales
+ ,item iss
+ ,date_dim d1
+ where ss_item_sk = iss.i_item_sk
+   and ss_sold_date_sk = d1.d_date_sk
+   and d1.d_year between 1999 AND 1999 + 2
+ intersect 
+ select ics.i_brand_id
+ ,ics.i_class_id
+ ,ics.i_category_id
+ from catalog_sales
+ ,item ics
+ ,date_dim d2
+ where cs_item_sk = ics.i_ite

[09/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query61.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query61.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query61.q.out
new file mode 100644
index 000..0b4f5fd
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query61.q.out
@@ -0,0 +1,280 @@
+Warning: Shuffle Join MERGEJOIN[156][tables = [$hdt$_0, $hdt$_1]] in Stage 
'Reducer 5' is a cross product
+PREHOOK: query: explain
+select  promotions,total,cast(promotions as decimal(15,4))/cast(total as 
decimal(15,4))*100
+from
+  (select sum(ss_ext_sales_price) promotions
+   from  store_sales
+,store
+,promotion
+,date_dim
+,customer
+,customer_address 
+,item
+   where ss_sold_date_sk = d_date_sk
+   and   ss_store_sk = s_store_sk
+   and   ss_promo_sk = p_promo_sk
+   and   ss_customer_sk= c_customer_sk
+   and   ca_address_sk = c_current_addr_sk
+   and   ss_item_sk = i_item_sk 
+   and   ca_gmt_offset = -7
+   and   i_category = 'Electronics'
+   and   (p_channel_dmail = 'Y' or p_channel_email = 'Y' or p_channel_tv = 'Y')
+   and   s_gmt_offset = -7
+   and   d_year = 1999
+   and   d_moy  = 11) promotional_sales,
+  (select sum(ss_ext_sales_price) total
+   from  store_sales
+,store
+,date_dim
+,customer
+,customer_address
+,item
+   where ss_sold_date_sk = d_date_sk
+   and   ss_store_sk = s_store_sk
+   and   ss_customer_sk= c_customer_sk
+   and   ca_address_sk = c_current_addr_sk
+   and   ss_item_sk = i_item_sk
+   and   ca_gmt_offset = -7
+   and   i_category = 'Electronics'
+   and   s_gmt_offset = -7
+   and   d_year = 1999
+   and   d_moy  = 11) all_sales
+order by promotions, total
+limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select  promotions,total,cast(promotions as decimal(15,4))/cast(total as 
decimal(15,4))*100
+from
+  (select sum(ss_ext_sales_price) promotions
+   from  store_sales
+,store
+,promotion
+,date_dim
+,customer
+,customer_address 
+,item
+   where ss_sold_date_sk = d_date_sk
+   and   ss_store_sk = s_store_sk
+   and   ss_promo_sk = p_promo_sk
+   and   ss_customer_sk= c_customer_sk
+   and   ca_address_sk = c_current_addr_sk
+   and   ss_item_sk = i_item_sk 
+   and   ca_gmt_offset = -7
+   and   i_category = 'Electronics'
+   and   (p_channel_dmail = 'Y' or p_channel_email = 'Y' or p_channel_tv = 'Y')
+   and   s_gmt_offset = -7
+   and   d_year = 1999
+   and   d_moy  = 11) promotional_sales,
+  (select sum(ss_ext_sales_price) total
+   from  store_sales
+,store
+,date_dim
+,customer
+,customer_address
+,item
+   where ss_sold_date_sk = d_date_sk
+   and   ss_store_sk = s_store_sk
+   and   ss_customer_sk= c_customer_sk
+   and   ca_address_sk = c_current_addr_sk
+   and   ss_item_sk = i_item_sk
+   and   ca_gmt_offset = -7
+   and   i_category = 'Electronics'
+   and   s_gmt_offset = -7
+   and   d_year = 1999
+   and   d_moy  = 11) all_sales
+order by promotions, total
+limit 100
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 12 <- Map 11 (SIMPLE_EDGE), Map 16 (SIMPLE_EDGE)
+Reducer 13 <- Map 20 (SIMPLE_EDGE), Reducer 12 (SIMPLE_EDGE)
+Reducer 14 <- Map 21 (SIMPLE_EDGE), Reducer 13 (SIMPLE_EDGE)
+Reducer 15 <- Map 22 (SIMPLE_EDGE), Reducer 14 (SIMPLE_EDGE)
+Reducer 17 <- Map 16 (SIMPLE_EDGE), Map 23 (SIMPLE_EDGE)
+Reducer 18 <- Map 20 (SIMPLE_EDGE), Reducer 17 (SIMPLE_EDGE)
+Reducer 19 <- Map 21 (SIMPLE_EDGE), Reducer 18 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 10 (SIMPLE_EDGE)
+Reducer 3 <- Reducer 15 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Reducer 3 (CUSTOM_SIMPLE_EDGE)
+Reducer 5 <- Reducer 4 (CUSTOM_SIMPLE_EDGE), Reducer 9 (CUSTOM_SIMPLE_EDGE)
+Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
+Reducer 7 <- Map 1 (SIMPLE_EDGE), Map 10 (SIMPLE_EDGE)
+Reducer 8 <- Reducer 19 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
+Reducer 9 <- Reducer 8 (CUSTOM_SIMPLE_EDGE)
+
+Stage-0
+  Fetch Operator
+limit:100
+Stage-1
+  Reducer 6
+  File Output Operator [FS_91]
+Limit [LIM_90] (rows=1 width=225)
+  Number of rows:100
+  Select Operator [SEL_89] (rows=1 width=225)
+Output:["_col0","_col1","_col2"]
+  <-Reducer 5 [SIMPLE_EDGE]
+SHUFFLE [RS_88]
+  Select Operator [SEL_87] (rows=1 width=225)
+Output:["_col0","_col1","_col2"]
+Merge Join Operator [MERGEJOIN_156] (rows=1 width=225)
+  Conds:(Inner),Output:["_col0","_col1"]
+<-Reducer 4 [CUSTOM_SIMPLE_EDGE]
+  PARTITION_ONLY_SHUFFLE [RS_84]
+Group By Operator [GBY_43] (rows=1 width=112)
+  Output:["_col0"],aggregations:["sum(VALUE._co

[36/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query27.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query27.q.out 
b/ql/src/test/results/clientpositive/perf/query27.q.out
deleted file mode 100644
index c6a1905..000
--- a/ql/src/test/results/clientpositive/perf/query27.q.out
+++ /dev/null
@@ -1,140 +0,0 @@
-PREHOOK: query: explain
-select  i_item_id,
-s_state, grouping(s_state) g_state,
-avg(ss_quantity) agg1,
-avg(ss_list_price) agg2,
-avg(ss_coupon_amt) agg3,
-avg(ss_sales_price) agg4
- from store_sales, customer_demographics, date_dim, store, item
- where ss_sold_date_sk = d_date_sk and
-   ss_item_sk = i_item_sk and
-   ss_store_sk = s_store_sk and
-   ss_cdemo_sk = cd_demo_sk and
-   cd_gender = 'M' and
-   cd_marital_status = 'U' and
-   cd_education_status = '2 yr Degree' and
-   d_year = 2001 and
-   s_state in ('SD','FL', 'MI', 'LA', 'MO', 'SC')
- group by rollup (i_item_id, s_state)
- order by i_item_id
- ,s_state
- limit 100
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-select  i_item_id,
-s_state, grouping(s_state) g_state,
-avg(ss_quantity) agg1,
-avg(ss_list_price) agg2,
-avg(ss_coupon_amt) agg3,
-avg(ss_sales_price) agg4
- from store_sales, customer_demographics, date_dim, store, item
- where ss_sold_date_sk = d_date_sk and
-   ss_item_sk = i_item_sk and
-   ss_store_sk = s_store_sk and
-   ss_cdemo_sk = cd_demo_sk and
-   cd_gender = 'M' and
-   cd_marital_status = 'U' and
-   cd_education_status = '2 yr Degree' and
-   d_year = 2001 and
-   s_state in ('SD','FL', 'MI', 'LA', 'MO', 'SC')
- group by rollup (i_item_id, s_state)
- order by i_item_id
- ,s_state
- limit 100
-POSTHOOK: type: QUERY
-Plan optimized by CBO.
-
-Vertex dependency in root stage
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 8 (SIMPLE_EDGE)
-Reducer 3 <- Map 9 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Map 10 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
-Reducer 5 <- Map 11 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
-Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
-Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
-
-Stage-0
-  Fetch Operator
-limit:100
-Stage-1
-  Reducer 7
-  File Output Operator [FS_36]
-Limit [LIM_35] (rows=100 width=88)
-  Number of rows:100
-  Select Operator [SEL_34] (rows=1264972921 width=88)
-Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"]
-  <-Reducer 6 [SIMPLE_EDGE]
-SHUFFLE [RS_33]
-  Select Operator [SEL_32] (rows=1264972921 width=88)
-
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"]
-Group By Operator [GBY_31] (rows=1264972921 width=88)
-  
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"],aggregations:["avg(VALUE._col0)","avg(VALUE._col1)","avg(VALUE._col2)","avg(VALUE._col3)"],keys:KEY._col0,
 KEY._col1, KEY._col2
-<-Reducer 5 [SIMPLE_EDGE]
-  SHUFFLE [RS_30]
-PartitionCols:_col0, _col1, _col2
-Group By Operator [GBY_29] (rows=2529945843 width=88)
-  
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"],aggregations:["avg(_col2)","avg(_col3)","avg(_col4)","avg(_col5)"],keys:_col0,
 _col1, 0
-  Select Operator [SEL_27] (rows=843315281 width=88)
-
Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
-Merge Join Operator [MERGEJOIN_59] (rows=843315281 
width=88)
-  
Conds:RS_24._col1=RS_25._col0(Inner),Output:["_col4","_col5","_col6","_col7","_col15","_col17"]
-<-Map 11 [SIMPLE_EDGE]
-  SHUFFLE [RS_25]
-PartitionCols:_col0
-Select Operator [SEL_14] (rows=462000 width=1436)
-  Output:["_col0","_col1"]
-  Filter Operator [FIL_55] (rows=462000 width=1436)
-predicate:i_item_sk is not null
-TableScan [TS_12] (rows=462000 width=1436)
-  
default@item,item,Tbl:COMPLETE,Col:NONE,Output:["i_item_sk","i_item_id"]
-<-Reducer 4 [SIMPLE_EDGE]
-  SHUFFLE [RS_24]
-PartitionCols:_col1
-Merge Join Operator [MERGEJOIN_58] (rows=766650239 
width=88)
-  
Conds:RS_21._col3=RS_22._col0(Inner),Output:["_col1","_col4","_col5","_col6","_col7","_col15"]
-<-Map 10 [SIMPLE_EDGE]
-  SHUFFLE [RS_22]
-PartitionCols:

[11/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query54.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query54.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query54.q.out
new file mode 100644
index 000..e6b9b28
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query54.q.out
@@ -0,0 +1,386 @@
+Warning: Shuffle Join MERGEJOIN[184][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3]] in Stage 'Reducer 4' is a cross product
+Warning: Shuffle Join MERGEJOIN[185][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4]] in Stage 'Reducer 5' is a cross product
+Warning: Shuffle Join MERGEJOIN[183][tables = [$hdt$_1, $hdt$_2]] in Stage 
'Reducer 30' is a cross product
+Warning: Shuffle Join MERGEJOIN[186][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in 
Stage 'Reducer 6' is a cross product
+PREHOOK: query: explain
+with my_customers as (
+ select distinct c_customer_sk
+, c_current_addr_sk
+ from   
+( select cs_sold_date_sk sold_date_sk,
+ cs_bill_customer_sk customer_sk,
+ cs_item_sk item_sk
+  from   catalog_sales
+  union all
+  select ws_sold_date_sk sold_date_sk,
+ ws_bill_customer_sk customer_sk,
+ ws_item_sk item_sk
+  from   web_sales
+ ) cs_or_ws_sales,
+ item,
+ date_dim,
+ customer
+ where   sold_date_sk = d_date_sk
+ and item_sk = i_item_sk
+ and i_category = 'Jewelry'
+ and i_class = 'consignment'
+ and c_customer_sk = cs_or_ws_sales.customer_sk
+ and d_moy = 3
+ and d_year = 1999
+ )
+ , my_revenue as (
+ select c_customer_sk,
+sum(ss_ext_sales_price) as revenue
+ from   my_customers,
+store_sales,
+customer_address,
+store,
+date_dim
+ where  c_current_addr_sk = ca_address_sk
+and ca_county = s_county
+and ca_state = s_state
+and ss_sold_date_sk = d_date_sk
+and c_customer_sk = ss_customer_sk
+and d_month_seq between (select distinct d_month_seq+1
+ from   date_dim where d_year = 1999 and d_moy 
= 3)
+   and  (select distinct d_month_seq+3
+ from   date_dim where d_year = 1999 and d_moy 
= 3)
+ group by c_customer_sk
+ )
+ , segments as
+ (select cast((revenue/50) as int) as segment
+  from   my_revenue
+ )
+  select  segment, count(*) as num_customers, segment*50 as segment_base
+ from segments
+ group by segment
+ order by segment, num_customers
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with my_customers as (
+ select distinct c_customer_sk
+, c_current_addr_sk
+ from   
+( select cs_sold_date_sk sold_date_sk,
+ cs_bill_customer_sk customer_sk,
+ cs_item_sk item_sk
+  from   catalog_sales
+  union all
+  select ws_sold_date_sk sold_date_sk,
+ ws_bill_customer_sk customer_sk,
+ ws_item_sk item_sk
+  from   web_sales
+ ) cs_or_ws_sales,
+ item,
+ date_dim,
+ customer
+ where   sold_date_sk = d_date_sk
+ and item_sk = i_item_sk
+ and i_category = 'Jewelry'
+ and i_class = 'consignment'
+ and c_customer_sk = cs_or_ws_sales.customer_sk
+ and d_moy = 3
+ and d_year = 1999
+ )
+ , my_revenue as (
+ select c_customer_sk,
+sum(ss_ext_sales_price) as revenue
+ from   my_customers,
+store_sales,
+customer_address,
+store,
+date_dim
+ where  c_current_addr_sk = ca_address_sk
+and ca_county = s_county
+and ca_state = s_state
+and ss_sold_date_sk = d_date_sk
+and c_customer_sk = ss_customer_sk
+and d_month_seq between (select distinct d_month_seq+1
+ from   date_dim where d_year = 1999 and d_moy 
= 3)
+   and  (select distinct d_month_seq+3
+ from   date_dim where d_year = 1999 and d_moy 
= 3)
+ group by c_customer_sk
+ )
+ , segments as
+ (select cast((revenue/50) as int) as segment
+  from   my_revenue
+ )
+  select  segment, count(*) as num_customers, segment*50 as segment_base
+ from segments
+ group by segment
+ order by segment, num_customers
+ limit 100
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Map 15 <- Union 16 (CONTAINS)
+Map 21 <- Union 16 (CONTAINS)
+Reducer 12 <- Map 11 (SIMPLE_EDGE), Map 14 (SIMPLE_EDGE)
+Reducer 13 <- Reducer 12 (SIMPLE_EDGE), Reducer 20 (SIMPLE_EDGE)
+Reducer 17 <- Map 22 (SIMPLE_EDGE), Union 16 (SIMPLE_EDGE)
+Reducer 18 <- Map 23 (SIMPLE_EDGE), Reducer 17 (SIMPLE_EDGE)
+Reducer 19 <- Map 24 (SIMPLE_EDGE), Reducer 18 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE

[35/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query33.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query33.q.out 
b/ql/src/test/results/clientpositive/perf/query33.q.out
deleted file mode 100644
index 0810c28..000
--- a/ql/src/test/results/clientpositive/perf/query33.q.out
+++ /dev/null
@@ -1,393 +0,0 @@
-PREHOOK: query: explain
-with ss as (
- select
-  i_manufact_id,sum(ss_ext_sales_price) total_sales
- from
-   store_sales,
-   date_dim,
- customer_address,
- item
- where
- i_manufact_id in (select
-  i_manufact_id
-from
- item
-where i_category in ('Books'))
- and ss_item_sk  = i_item_sk
- and ss_sold_date_sk = d_date_sk
- and d_year  = 1999
- and d_moy   = 3
- and ss_addr_sk  = ca_address_sk
- and ca_gmt_offset   = -6 
- group by i_manufact_id),
- cs as (
- select
-  i_manufact_id,sum(cs_ext_sales_price) total_sales
- from
-   catalog_sales,
-   date_dim,
- customer_address,
- item
- where
- i_manufact_id   in (select
-  i_manufact_id
-from
- item
-where i_category in ('Books'))
- and cs_item_sk  = i_item_sk
- and cs_sold_date_sk = d_date_sk
- and d_year  = 1999
- and d_moy   = 3
- and cs_bill_addr_sk = ca_address_sk
- and ca_gmt_offset   = -6 
- group by i_manufact_id),
- ws as (
- select
-  i_manufact_id,sum(ws_ext_sales_price) total_sales
- from
-   web_sales,
-   date_dim,
- customer_address,
- item
- where
- i_manufact_id   in (select
-  i_manufact_id
-from
- item
-where i_category in ('Books'))
- and ws_item_sk  = i_item_sk
- and ws_sold_date_sk = d_date_sk
- and d_year  = 1999
- and d_moy   = 3
- and ws_bill_addr_sk = ca_address_sk
- and ca_gmt_offset   = -6
- group by i_manufact_id)
-  select  i_manufact_id ,sum(total_sales) total_sales
- from  (select * from ss 
-union all
-select * from cs 
-union all
-select * from ws) tmp1
- group by i_manufact_id
- order by total_sales
-limit 100
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-with ss as (
- select
-  i_manufact_id,sum(ss_ext_sales_price) total_sales
- from
-   store_sales,
-   date_dim,
- customer_address,
- item
- where
- i_manufact_id in (select
-  i_manufact_id
-from
- item
-where i_category in ('Books'))
- and ss_item_sk  = i_item_sk
- and ss_sold_date_sk = d_date_sk
- and d_year  = 1999
- and d_moy   = 3
- and ss_addr_sk  = ca_address_sk
- and ca_gmt_offset   = -6 
- group by i_manufact_id),
- cs as (
- select
-  i_manufact_id,sum(cs_ext_sales_price) total_sales
- from
-   catalog_sales,
-   date_dim,
- customer_address,
- item
- where
- i_manufact_id   in (select
-  i_manufact_id
-from
- item
-where i_category in ('Books'))
- and cs_item_sk  = i_item_sk
- and cs_sold_date_sk = d_date_sk
- and d_year  = 1999
- and d_moy   = 3
- and cs_bill_addr_sk = ca_address_sk
- and ca_gmt_offset   = -6 
- group by i_manufact_id),
- ws as (
- select
-  i_manufact_id,sum(ws_ext_sales_price) total_sales
- from
-   web_sales,
-   date_dim,
- customer_address,
- item
- where
- i_manufact_id   in (select
-  i_manufact_id
-from
- item
-where i_category in ('Books'))
- and ws_item_sk  = i_item_sk
- and ws_sold_date_sk = d_date_sk
- and d_year  = 1999
- and d_moy   = 3
- and ws_bill_addr_sk = ca_address_sk
- and ca_gmt_offset   = -6
- group by i_manufact_id)
-  select  i_manufact_id ,sum(total_sales) total_sales
- from  (select * from ss 
-union all
-select * from cs 
-union all
-select * from ws) tmp1
- group by i_manufact_id
- order by total_sales
-limit 100
-POSTHOOK: type: QUERY
-Plan optimized by CBO.
-
-Vertex dependency in root stage
-Reducer 10 <- Reducer 9 (SIMPLE_EDGE), Union 5 (CONTAINS)
-Reducer 11 <- Map 1 (SIMPLE_EDGE), Reducer 17 (ONE_TO_ONE_EDGE)
-Reducer 12 <- Reducer 11 (SIMPLE_EDGE), Reducer 25 (SIMPLE_EDGE)
-Reducer 13 <- Reducer 12 (SIMPLE_EDGE), Union 5 (CONTAINS)
-Reducer 15 <- Map 14 (SIMPLE_EDGE)
-Reducer 16 <- Map 14 (SIMPLE_EDGE)
-Reducer 17 <- Map 14 (SIMPLE_EDGE)
-Reducer 19 <- Map 18 (SIMPLE_EDGE), Map 21 (SIMPLE_EDGE)
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 15 (ONE_TO_ONE_EDGE)
-Reducer

[41/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) 
(addendum)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/9244fdc7
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/9244fdc7
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/9244fdc7

Branch: refs/heads/master
Commit: 9244fdc7e1035c69ec41f0131f3b41027dabfda0
Parents: c9f5605
Author: Sahil Takiar 
Authored: Fri Sep 29 22:40:52 2017 -0700
Committer: Sahil Takiar 
Committed: Fri Sep 29 22:40:52 2017 -0700

--
 data/conf/perf-reg/hive-site.xml|  285 
 data/conf/perf-reg/tez-site.xml |   10 -
 data/conf/perf-reg/tez/hive-site.xml|  285 
 data/conf/perf-reg/tez/tez-site.xml |   10 +
 .../results/clientpositive/perf/query1.q.out|  169 ---
 .../results/clientpositive/perf/query10.q.out   |  288 
 .../results/clientpositive/perf/query11.q.out   |  351 -
 .../results/clientpositive/perf/query12.q.out   |  138 --
 .../results/clientpositive/perf/query13.q.out   |  205 ---
 .../results/clientpositive/perf/query14.q.out   | 1286 --
 .../results/clientpositive/perf/query15.q.out   |  119 --
 .../results/clientpositive/perf/query16.q.out   |  192 ---
 .../results/clientpositive/perf/query17.q.out   |  225 ---
 .../results/clientpositive/perf/query18.q.out   |  194 ---
 .../results/clientpositive/perf/query19.q.out   |  161 ---
 .../results/clientpositive/perf/query2.q.out|  255 
 .../results/clientpositive/perf/query20.q.out   |  130 --
 .../results/clientpositive/perf/query21.q.out   |  139 --
 .../results/clientpositive/perf/query22.q.out   |  121 --
 .../results/clientpositive/perf/query23.q.out   |  556 
 .../results/clientpositive/perf/query24.q.out   |  290 
 .../results/clientpositive/perf/query25.q.out   |  229 
 .../results/clientpositive/perf/query26.q.out   |  132 --
 .../results/clientpositive/perf/query27.q.out   |  140 --
 .../results/clientpositive/perf/query28.q.out   |  256 
 .../results/clientpositive/perf/query29.q.out   |  230 
 .../results/clientpositive/perf/query3.q.out|  102 --
 .../results/clientpositive/perf/query30.q.out   |  211 ---
 .../results/clientpositive/perf/query31.q.out   |  358 -
 .../results/clientpositive/perf/query32.q.out   |  144 --
 .../results/clientpositive/perf/query33.q.out   |  393 --
 .../results/clientpositive/perf/query34.q.out   |  156 ---
 .../results/clientpositive/perf/query35.q.out   |  284 
 .../results/clientpositive/perf/query36.q.out   |  149 --
 .../results/clientpositive/perf/query37.q.out   |  103 --
 .../results/clientpositive/perf/query38.q.out   |  212 ---
 .../results/clientpositive/perf/query39.q.out   |  255 
 .../results/clientpositive/perf/query4.q.out|  500 ---
 .../results/clientpositive/perf/query40.q.out   |  148 --
 .../results/clientpositive/perf/query42.q.out   |  108 --
 .../results/clientpositive/perf/query43.q.out   |  100 --
 .../results/clientpositive/perf/query44.q.out   |  258 
 .../results/clientpositive/perf/query45.q.out   |  180 ---
 .../results/clientpositive/perf/query46.q.out   |  191 ---
 .../results/clientpositive/perf/query47.q.out   |  325 -
 .../results/clientpositive/perf/query48.q.out   |  218 ---
 .../results/clientpositive/perf/query49.q.out   |  496 ---
 .../results/clientpositive/perf/query5.q.out|  473 ---
 .../results/clientpositive/perf/query50.q.out   |  210 ---
 .../results/clientpositive/perf/query51.q.out   |  188 ---
 .../results/clientpositive/perf/query52.q.out   |  106 --
 .../results/clientpositive/perf/query53.q.out   |  141 --
 .../results/clientpositive/perf/query54.q.out   |  386 --
 .../results/clientpositive/perf/query55.q.out   |   88 --
 .../results/clientpositive/perf/query56.q.out   |  379 --
 .../results/clientpositive/perf/query57.q.out   |  319 -
 .../results/clientpositive/perf/query58.q.out   |  440 --
 .../results/clientpositive/perf/query59.q.out   |  230 
 .../results/clientpositive/perf/query6.q.out|  226 ---
 .../results/clientpositive/perf/query60.q.out   |  399 --
 .../results/clientpositive/perf/query61.q.out   |  280 
 .../results/clientpositive/perf/query63.q.out   |  143 --
 .../results/clientpositive/perf/query64.q.out   |  772 ---
 .../results/clientpositive/perf/query65.q.out   |  169 ---
 .../results/clientpositive/perf/query66.q.out   |  612 -
 .../results/clientpositive/perf/query67.q.out   |  179 ---
 .../results/clientpositive/perf/query68.q.out   |  205 ---
 .../results/clientpositive/perf/query69.q.out   |  268 
 .../results/clientpositive/perf/query7.q.out|  132 --
 .../results/clientpositive/perf/query70.q.out   |  205 ---
 .../results/clientpositive/perf/query71.q.out   |  208 ---
 .../results/clientpositive/perf/que

[23/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query87.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query87.q.out 
b/ql/src/test/results/clientpositive/perf/query87.q.out
deleted file mode 100644
index 8430e8a..000
--- a/ql/src/test/results/clientpositive/perf/query87.q.out
+++ /dev/null
@@ -1,239 +0,0 @@
-PREHOOK: query: explain
-select count(*) 
-from ((select distinct c_last_name, c_first_name, d_date
-   from store_sales, date_dim, customer
-   where store_sales.ss_sold_date_sk = date_dim.d_date_sk
- and store_sales.ss_customer_sk = customer.c_customer_sk
- and d_month_seq between 1212 and 1212+11)
-   except
-  (select distinct c_last_name, c_first_name, d_date
-   from catalog_sales, date_dim, customer
-   where catalog_sales.cs_sold_date_sk = date_dim.d_date_sk
- and catalog_sales.cs_bill_customer_sk = customer.c_customer_sk
- and d_month_seq between 1212 and 1212+11)
-   except
-  (select distinct c_last_name, c_first_name, d_date
-   from web_sales, date_dim, customer
-   where web_sales.ws_sold_date_sk = date_dim.d_date_sk
- and web_sales.ws_bill_customer_sk = customer.c_customer_sk
- and d_month_seq between 1212 and 1212+11)
-) cool_cust
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-select count(*) 
-from ((select distinct c_last_name, c_first_name, d_date
-   from store_sales, date_dim, customer
-   where store_sales.ss_sold_date_sk = date_dim.d_date_sk
- and store_sales.ss_customer_sk = customer.c_customer_sk
- and d_month_seq between 1212 and 1212+11)
-   except
-  (select distinct c_last_name, c_first_name, d_date
-   from catalog_sales, date_dim, customer
-   where catalog_sales.cs_sold_date_sk = date_dim.d_date_sk
- and catalog_sales.cs_bill_customer_sk = customer.c_customer_sk
- and d_month_seq between 1212 and 1212+11)
-   except
-  (select distinct c_last_name, c_first_name, d_date
-   from web_sales, date_dim, customer
-   where web_sales.ws_sold_date_sk = date_dim.d_date_sk
- and web_sales.ws_bill_customer_sk = customer.c_customer_sk
- and d_month_seq between 1212 and 1212+11)
-) cool_cust
-POSTHOOK: type: QUERY
-Plan optimized by CBO.
-
-Vertex dependency in root stage
-Reducer 11 <- Map 10 (SIMPLE_EDGE), Map 18 (SIMPLE_EDGE)
-Reducer 12 <- Map 17 (SIMPLE_EDGE), Reducer 11 (SIMPLE_EDGE)
-Reducer 13 <- Reducer 12 (SIMPLE_EDGE), Union 5 (CONTAINS)
-Reducer 14 <- Map 10 (SIMPLE_EDGE), Map 19 (SIMPLE_EDGE)
-Reducer 15 <- Map 17 (SIMPLE_EDGE), Reducer 14 (SIMPLE_EDGE)
-Reducer 16 <- Reducer 15 (SIMPLE_EDGE), Union 7 (CONTAINS)
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 10 (SIMPLE_EDGE)
-Reducer 3 <- Map 17 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Reducer 3 (SIMPLE_EDGE), Union 5 (CONTAINS)
-Reducer 6 <- Union 5 (SIMPLE_EDGE), Union 7 (CONTAINS)
-Reducer 8 <- Union 7 (SIMPLE_EDGE)
-Reducer 9 <- Reducer 8 (CUSTOM_SIMPLE_EDGE)
-
-Stage-0
-  Fetch Operator
-limit:-1
-Stage-1
-  Reducer 9
-  File Output Operator [FS_104]
-Group By Operator [GBY_102] (rows=1 width=24)
-  Output:["_col0"],aggregations:["count(VALUE._col0)"]
-<-Reducer 8 [CUSTOM_SIMPLE_EDGE]
-  PARTITION_ONLY_SHUFFLE [RS_101]
-Group By Operator [GBY_100] (rows=1 width=24)
-  Output:["_col0"],aggregations:["count()"]
-  Select Operator [SEL_98] (rows=4537552 width=129)
-Filter Operator [FIL_97] (rows=4537552 width=129)
-  predicate:(((_col3 * 2) = _col4) and (_col3 > 0))
-  Select Operator [SEL_129] (rows=27225312 width=129)
-Output:["_col3","_col4"]
-Group By Operator [GBY_96] (rows=27225312 width=129)
-  
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["sum(VALUE._col0)","sum(VALUE._col1)"],keys:KEY._col0,
 KEY._col1, KEY._col2
-<-Union 7 [SIMPLE_EDGE]
-  <-Reducer 16 [CONTAINS]
-Reduce Output Operator [RS_95]
-  PartitionCols:_col0, _col1, _col2
-  Group By Operator [GBY_94] (rows=54450625 width=129)
-
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["sum(_col3)","sum(_col4)"],keys:_col0,
 _col1, _col2
-Select Operator [SEL_92] (rows=54450625 width=129)
-  Output:["_col0","_col1","_col2","_col3","_col4"]
-  Select Operator [SEL_89] (rows=43560808 
width=135)
-
Output:["_col0","_col1","_col2","_col3","_col4"]
-Group By Operator [GBY_88] (rows=43560808 
width=135)
-  
Output:["_

[01/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

Repository: hive
Updated Branches:
  refs/heads/master c9f56052c -> 9244fdc7e


http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query95.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query95.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query95.q.out
new file mode 100644
index 000..dd3e344
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query95.q.out
@@ -0,0 +1,231 @@
+PREHOOK: query: explain
+with ws_wh as
+(select ws1.ws_order_number,ws1.ws_warehouse_sk wh1,ws2.ws_warehouse_sk wh2
+ from web_sales ws1,web_sales ws2
+ where ws1.ws_order_number = ws2.ws_order_number
+   and ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk)
+ select  
+   count(distinct ws_order_number) as `order count`
+  ,sum(ws_ext_ship_cost) as `total shipping cost`
+  ,sum(ws_net_profit) as `total net profit`
+from
+   web_sales ws1
+  ,date_dim
+  ,customer_address
+  ,web_site
+where
+d_date between '1999-5-01' and 
+   (cast('1999-5-01' as date) + 60 days)
+and ws1.ws_ship_date_sk = d_date_sk
+and ws1.ws_ship_addr_sk = ca_address_sk
+and ca_state = 'TX'
+and ws1.ws_web_site_sk = web_site_sk
+and web_company_name = 'pri'
+and ws1.ws_order_number in (select ws_order_number
+from ws_wh)
+and ws1.ws_order_number in (select wr_order_number
+from web_returns,ws_wh
+where wr_order_number = ws_wh.ws_order_number)
+order by count(distinct ws_order_number)
+limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with ws_wh as
+(select ws1.ws_order_number,ws1.ws_warehouse_sk wh1,ws2.ws_warehouse_sk wh2
+ from web_sales ws1,web_sales ws2
+ where ws1.ws_order_number = ws2.ws_order_number
+   and ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk)
+ select  
+   count(distinct ws_order_number) as `order count`
+  ,sum(ws_ext_ship_cost) as `total shipping cost`
+  ,sum(ws_net_profit) as `total net profit`
+from
+   web_sales ws1
+  ,date_dim
+  ,customer_address
+  ,web_site
+where
+d_date between '1999-5-01' and 
+   (cast('1999-5-01' as date) + 60 days)
+and ws1.ws_ship_date_sk = d_date_sk
+and ws1.ws_ship_addr_sk = ca_address_sk
+and ca_state = 'TX'
+and ws1.ws_web_site_sk = web_site_sk
+and web_company_name = 'pri'
+and ws1.ws_order_number in (select ws_order_number
+from ws_wh)
+and ws1.ws_order_number in (select wr_order_number
+from web_returns,ws_wh
+where wr_order_number = ws_wh.ws_order_number)
+order by count(distinct ws_order_number)
+limit 100
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 13 <- Map 12 (SIMPLE_EDGE), Map 18 (SIMPLE_EDGE)
+Reducer 14 <- Map 19 (SIMPLE_EDGE), Reducer 13 (SIMPLE_EDGE)
+Reducer 15 <- Reducer 14 (SIMPLE_EDGE)
+Reducer 16 <- Map 12 (SIMPLE_EDGE), Map 18 (SIMPLE_EDGE)
+Reducer 17 <- Reducer 16 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 9 (SIMPLE_EDGE)
+Reducer 3 <- Map 10 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Map 11 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
+Reducer 5 <- Reducer 15 (ONE_TO_ONE_EDGE), Reducer 17 (ONE_TO_ONE_EDGE), 
Reducer 4 (SIMPLE_EDGE)
+Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
+Reducer 7 <- Reducer 6 (CUSTOM_SIMPLE_EDGE)
+Reducer 8 <- Reducer 7 (SIMPLE_EDGE)
+
+Stage-0
+  Fetch Operator
+limit:-1
+Stage-1
+  Reducer 8
+  File Output Operator [FS_71]
+Limit [LIM_69] (rows=1 width=344)
+  Number of rows:100
+  Select Operator [SEL_68] (rows=1 width=344)
+Output:["_col0","_col1","_col2"]
+  <-Reducer 7 [SIMPLE_EDGE]
+SHUFFLE [RS_67]
+  Select Operator [SEL_66] (rows=1 width=344)
+Output:["_col1","_col2","_col3"]
+Group By Operator [GBY_120] (rows=1 width=344)
+  
Output:["_col0","_col1","_col2"],aggregations:["count(VALUE._col0)","sum(VALUE._col1)","sum(VALUE._col2)"]
+<-Reducer 6 [CUSTOM_SIMPLE_EDGE]
+  PARTITION_ONLY_SHUFFLE [RS_119]
+Group By Operator [GBY_118] (rows=1 width=344)
+  
Output:["_col0","_col1","_col2"],aggregations:["count(_col0)","sum(_col1)","sum(_col2)"]
+  Group By Operator [GBY_117] (rows=421668645 width=135)
+
Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)","sum(VALUE._col1)"],keys:KEY._col0
+  <-Reducer 5 [SIMPLE_EDGE]
+SHUFFLE [RS_116]
+  PartitionCols:_col0
+  Group By Operator [GBY_115] (rows=421668645 
width=135)
+
Output:["_col0","_col2","_col3"],aggregations:["sum(_col4)","sum(_col5)"],keys:_col3
+Merge Join Operator [MERGEJOIN_127] 
(rows=421668645 widt

[26/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query75.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query75.q.out 
b/ql/src/test/results/clientpositive/perf/query75.q.out
deleted file mode 100644
index 08e1620..000
--- a/ql/src/test/results/clientpositive/perf/query75.q.out
+++ /dev/null
@@ -1,481 +0,0 @@
-PREHOOK: query: explain
-WITH all_sales AS (
- SELECT d_year
-   ,i_brand_id
-   ,i_class_id
-   ,i_category_id
-   ,i_manufact_id
-   ,SUM(sales_cnt) AS sales_cnt
-   ,SUM(sales_amt) AS sales_amt
- FROM (SELECT d_year
- ,i_brand_id
- ,i_class_id
- ,i_category_id
- ,i_manufact_id
- ,cs_quantity - COALESCE(cr_return_quantity,0) AS sales_cnt
- ,cs_ext_sales_price - COALESCE(cr_return_amount,0.0) AS sales_amt
-   FROM catalog_sales JOIN item ON i_item_sk=cs_item_sk
-  JOIN date_dim ON d_date_sk=cs_sold_date_sk
-  LEFT JOIN catalog_returns ON 
(cs_order_number=cr_order_number 
-AND cs_item_sk=cr_item_sk)
-   WHERE i_category='Sports'
-   UNION
-   SELECT d_year
- ,i_brand_id
- ,i_class_id
- ,i_category_id
- ,i_manufact_id
- ,ss_quantity - COALESCE(sr_return_quantity,0) AS sales_cnt
- ,ss_ext_sales_price - COALESCE(sr_return_amt,0.0) AS sales_amt
-   FROM store_sales JOIN item ON i_item_sk=ss_item_sk
-JOIN date_dim ON d_date_sk=ss_sold_date_sk
-LEFT JOIN store_returns ON 
(ss_ticket_number=sr_ticket_number 
-AND ss_item_sk=sr_item_sk)
-   WHERE i_category='Sports'
-   UNION
-   SELECT d_year
- ,i_brand_id
- ,i_class_id
- ,i_category_id
- ,i_manufact_id
- ,ws_quantity - COALESCE(wr_return_quantity,0) AS sales_cnt
- ,ws_ext_sales_price - COALESCE(wr_return_amt,0.0) AS sales_amt
-   FROM web_sales JOIN item ON i_item_sk=ws_item_sk
-  JOIN date_dim ON d_date_sk=ws_sold_date_sk
-  LEFT JOIN web_returns ON 
(ws_order_number=wr_order_number 
-AND ws_item_sk=wr_item_sk)
-   WHERE i_category='Sports') sales_detail
- GROUP BY d_year, i_brand_id, i_class_id, i_category_id, i_manufact_id)
- SELECT  prev_yr.d_year AS prev_year
-  ,curr_yr.d_year AS year
-  ,curr_yr.i_brand_id
-  ,curr_yr.i_class_id
-  ,curr_yr.i_category_id
-  ,curr_yr.i_manufact_id
-  ,prev_yr.sales_cnt AS prev_yr_cnt
-  ,curr_yr.sales_cnt AS curr_yr_cnt
-  ,curr_yr.sales_cnt-prev_yr.sales_cnt AS 
sales_cnt_diff
-  ,curr_yr.sales_amt-prev_yr.sales_amt AS 
sales_amt_diff
- FROM all_sales curr_yr, all_sales prev_yr
- WHERE curr_yr.i_brand_id=prev_yr.i_brand_id
-   AND curr_yr.i_class_id=prev_yr.i_class_id
-   AND curr_yr.i_category_id=prev_yr.i_category_id
-   AND curr_yr.i_manufact_id=prev_yr.i_manufact_id
-   AND curr_yr.d_year=2002
-   AND prev_yr.d_year=2002-1
-   AND CAST(curr_yr.sales_cnt AS DECIMAL(17,2))/CAST(prev_yr.sales_cnt AS 
DECIMAL(17,2))<0.9
- ORDER BY sales_cnt_diff
- limit 100
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-WITH all_sales AS (
- SELECT d_year
-   ,i_brand_id
-   ,i_class_id
-   ,i_category_id
-   ,i_manufact_id
-   ,SUM(sales_cnt) AS sales_cnt
-   ,SUM(sales_amt) AS sales_amt
- FROM (SELECT d_year
- ,i_brand_id
- ,i_class_id
- ,i_category_id
- ,i_manufact_id
- ,cs_quantity - COALESCE(cr_return_quantity,0) AS sales_cnt
- ,cs_ext_sales_price - COALESCE(cr_return_amount,0.0) AS sales_amt
-   FROM catalog_sales JOIN item ON i_item_sk=cs_item_sk
-  JOIN date_dim ON d_date_sk=cs_sold_date_sk
-  LEFT JOIN catalog_returns ON 
(cs_order_number=cr_order_number 
-AND cs_item_sk=cr_item_sk)
-   WHERE i_category='Sports'
-   UNION
-   SELECT d_year
- ,i_brand_id
- ,i_class_id
- ,i_category_id
- ,i_manufact_id
- ,ss_quantity - COALESCE(sr_return_quantity,0) AS sales_cnt
- ,ss_ext_sales_price - COALESCE(sr_return_amt,0.0) AS sales_amt
-   FROM store_sales JOIN item ON i_item_sk=ss_item_sk
-JOIN date_dim ON d_date_sk=ss_sold_date_sk
-LEFT JOIN store_returns ON 
(ss_ticket_number=sr_ticket_number 
-

[28/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query65.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query65.q.out 
b/ql/src/test/results/clientpositive/perf/query65.q.out
deleted file mode 100644
index 0091ad0..000
--- a/ql/src/test/results/clientpositive/perf/query65.q.out
+++ /dev/null
@@ -1,169 +0,0 @@
-PREHOOK: query: explain
-select 
-   s_store_name,
-   i_item_desc,
-   sc.revenue,
-   i_current_price,
-   i_wholesale_cost,
-   i_brand
- from store, item,
- (select ss_store_sk, avg(revenue) as ave
-   from
-   (select  ss_store_sk, ss_item_sk, 
-sum(ss_sales_price) as revenue
-   from store_sales, date_dim
-   where ss_sold_date_sk = d_date_sk and d_month_seq between 1212 
and 1212+11
-   group by ss_store_sk, ss_item_sk) sa
-   group by ss_store_sk) sb,
- (select  ss_store_sk, ss_item_sk, sum(ss_sales_price) as revenue
-   from store_sales, date_dim
-   where ss_sold_date_sk = d_date_sk and d_month_seq between 1212 and 
1212+11
-   group by ss_store_sk, ss_item_sk) sc
- where sb.ss_store_sk = sc.ss_store_sk and 
-   sc.revenue <= 0.1 * sb.ave and
-   s_store_sk = sc.ss_store_sk and
-   i_item_sk = sc.ss_item_sk
- order by s_store_name, i_item_desc
-limit 100
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-select 
-   s_store_name,
-   i_item_desc,
-   sc.revenue,
-   i_current_price,
-   i_wholesale_cost,
-   i_brand
- from store, item,
- (select ss_store_sk, avg(revenue) as ave
-   from
-   (select  ss_store_sk, ss_item_sk, 
-sum(ss_sales_price) as revenue
-   from store_sales, date_dim
-   where ss_sold_date_sk = d_date_sk and d_month_seq between 1212 
and 1212+11
-   group by ss_store_sk, ss_item_sk) sa
-   group by ss_store_sk) sb,
- (select  ss_store_sk, ss_item_sk, sum(ss_sales_price) as revenue
-   from store_sales, date_dim
-   where ss_sold_date_sk = d_date_sk and d_month_seq between 1212 and 
1212+11
-   group by ss_store_sk, ss_item_sk) sc
- where sb.ss_store_sk = sc.ss_store_sk and 
-   sc.revenue <= 0.1 * sb.ave and
-   s_store_sk = sc.ss_store_sk and
-   i_item_sk = sc.ss_item_sk
- order by s_store_name, i_item_desc
-limit 100
-POSTHOOK: type: QUERY
-Plan optimized by CBO.
-
-Vertex dependency in root stage
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 9 (SIMPLE_EDGE)
-Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Map 10 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE), Reducer 8 
(SIMPLE_EDGE)
-Reducer 5 <- Map 11 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
-Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
-Reducer 7 <- Map 1 (SIMPLE_EDGE), Map 9 (SIMPLE_EDGE)
-Reducer 8 <- Reducer 7 (SIMPLE_EDGE)
-
-Stage-0
-  Fetch Operator
-limit:100
-Stage-1
-  Reducer 6
-  File Output Operator [FS_51]
-Limit [LIM_50] (rows=100 width=88)
-  Number of rows:100
-  Select Operator [SEL_49] (rows=20079 width=88)
-Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
-  <-Reducer 5 [SIMPLE_EDGE]
-SHUFFLE [RS_48]
-  Select Operator [SEL_47] (rows=20079 width=88)
-Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
-Merge Join Operator [MERGEJOIN_81] (rows=20079 width=88)
-  
Conds:RS_44._col1=RS_45._col0(Inner),Output:["_col2","_col6","_col8","_col9","_col10","_col11"]
-<-Map 11 [SIMPLE_EDGE]
-  SHUFFLE [RS_45]
-PartitionCols:_col0
-Select Operator [SEL_38] (rows=462000 width=1436)
-  Output:["_col0","_col1","_col2","_col3","_col4"]
-  Filter Operator [FIL_77] (rows=462000 width=1436)
-predicate:i_item_sk is not null
-TableScan [TS_36] (rows=462000 width=1436)
-  
default@item,item,Tbl:COMPLETE,Col:NONE,Output:["i_item_sk","i_item_desc","i_current_price","i_wholesale_cost","i_brand"]
-<-Reducer 4 [SIMPLE_EDGE]
-  SHUFFLE [RS_44]
-PartitionCols:_col1
-Filter Operator [FIL_43] (rows=232318249 width=88)
-  predicate:(_col2 <= (0.1 * _col4))
-  Merge Join Operator [MERGEJOIN_80] (rows=696954748 
width=88)
-
Conds:RS_39._col0=RS_40._col0(Inner),RS_39._col0=RS_41._col0(Inner),Output:["_col1","_col2","_col4","_col6"]
-  <-Map 10 [SIMPLE_EDGE]
-SHUFFLE [RS_41]
-  PartitionCols:_col0
-  Select Operator [SEL_35] (rows=1704 width=1910)
-Output:["_col0","_col1"]
-

[27/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query7.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query7.q.out 
b/ql/src/test/results/clientpositive/perf/query7.q.out
deleted file mode 100644
index 00628db..000
--- a/ql/src/test/results/clientpositive/perf/query7.q.out
+++ /dev/null
@@ -1,132 +0,0 @@
-PREHOOK: query: explain
-select  i_item_id, 
-avg(ss_quantity) agg1,
-avg(ss_list_price) agg2,
-avg(ss_coupon_amt) agg3,
-avg(ss_sales_price) agg4 
- from store_sales, customer_demographics, date_dim, item, promotion
- where ss_sold_date_sk = d_date_sk and
-   ss_item_sk = i_item_sk and
-   ss_cdemo_sk = cd_demo_sk and
-   ss_promo_sk = p_promo_sk and
-   cd_gender = 'F' and 
-   cd_marital_status = 'W' and
-   cd_education_status = 'Primary' and
-   (p_channel_email = 'N' or p_channel_event = 'N') and
-   d_year = 1998 
- group by i_item_id
- order by i_item_id
- limit 100
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-select  i_item_id, 
-avg(ss_quantity) agg1,
-avg(ss_list_price) agg2,
-avg(ss_coupon_amt) agg3,
-avg(ss_sales_price) agg4 
- from store_sales, customer_demographics, date_dim, item, promotion
- where ss_sold_date_sk = d_date_sk and
-   ss_item_sk = i_item_sk and
-   ss_cdemo_sk = cd_demo_sk and
-   ss_promo_sk = p_promo_sk and
-   cd_gender = 'F' and 
-   cd_marital_status = 'W' and
-   cd_education_status = 'Primary' and
-   (p_channel_email = 'N' or p_channel_event = 'N') and
-   d_year = 1998 
- group by i_item_id
- order by i_item_id
- limit 100
-POSTHOOK: type: QUERY
-Plan optimized by CBO.
-
-Vertex dependency in root stage
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 8 (SIMPLE_EDGE)
-Reducer 3 <- Map 9 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Map 10 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
-Reducer 5 <- Map 11 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
-Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
-Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
-
-Stage-0
-  Fetch Operator
-limit:100
-Stage-1
-  Reducer 7
-  File Output Operator [FS_35]
-Limit [LIM_34] (rows=100 width=88)
-  Number of rows:100
-  Select Operator [SEL_33] (rows=421657640 width=88)
-Output:["_col0","_col1","_col2","_col3","_col4"]
-  <-Reducer 6 [SIMPLE_EDGE]
-SHUFFLE [RS_32]
-  Group By Operator [GBY_30] (rows=421657640 width=88)
-
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["avg(VALUE._col0)","avg(VALUE._col1)","avg(VALUE._col2)","avg(VALUE._col3)"],keys:KEY._col0
-  <-Reducer 5 [SIMPLE_EDGE]
-SHUFFLE [RS_29]
-  PartitionCols:_col0
-  Group By Operator [GBY_28] (rows=843315281 width=88)
-
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["avg(_col4)","avg(_col5)","avg(_col7)","avg(_col6)"],keys:_col18
-Merge Join Operator [MERGEJOIN_58] (rows=843315281 
width=88)
-  
Conds:RS_24._col1=RS_25._col0(Inner),Output:["_col4","_col5","_col6","_col7","_col18"]
-<-Map 11 [SIMPLE_EDGE]
-  SHUFFLE [RS_25]
-PartitionCols:_col0
-Select Operator [SEL_14] (rows=462000 width=1436)
-  Output:["_col0","_col1"]
-  Filter Operator [FIL_54] (rows=462000 width=1436)
-predicate:i_item_sk is not null
-TableScan [TS_12] (rows=462000 width=1436)
-  
default@item,item,Tbl:COMPLETE,Col:NONE,Output:["i_item_sk","i_item_id"]
-<-Reducer 4 [SIMPLE_EDGE]
-  SHUFFLE [RS_24]
-PartitionCols:_col1
-Merge Join Operator [MERGEJOIN_57] (rows=766650239 
width=88)
-  
Conds:RS_21._col3=RS_22._col0(Inner),Output:["_col1","_col4","_col5","_col6","_col7"]
-<-Map 10 [SIMPLE_EDGE]
-  SHUFFLE [RS_22]
-PartitionCols:_col0
-Select Operator [SEL_11] (rows=2300 width=1179)
-  Output:["_col0"]
-  Filter Operator [FIL_53] (rows=2300 width=1179)
-predicate:(((p_channel_email = 'N') or 
(p_channel_event = 'N')) and p_promo_sk is not null)
-TableScan [TS_9] (rows=2300 width=1179)
-  
default@promotion,promotion,Tbl:COMPLETE,Col:NONE,Output:["p_promo_sk","p_channel_email","p_channel_event"]
-<-Reducer 3 [SIMPLE_EDGE]
-  SHUFFLE [RS_21]
-

[30/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query58.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query58.q.out 
b/ql/src/test/results/clientpositive/perf/query58.q.out
deleted file mode 100644
index d5ef23b..000
--- a/ql/src/test/results/clientpositive/perf/query58.q.out
+++ /dev/null
@@ -1,440 +0,0 @@
-Warning: Shuffle Join MERGEJOIN[265][tables = [$hdt$_4, $hdt$_5]] in Stage 
'Reducer 20' is a cross product
-Warning: Shuffle Join MERGEJOIN[269][tables = [$hdt$_5, $hdt$_6]] in Stage 
'Reducer 24' is a cross product
-Warning: Shuffle Join MERGEJOIN[273][tables = [$hdt$_6, $hdt$_7]] in Stage 
'Reducer 28' is a cross product
-PREHOOK: query: explain
-with ss_items as
- (select i_item_id item_id
-,sum(ss_ext_sales_price) ss_item_rev 
- from store_sales
- ,item
- ,date_dim
- where ss_item_sk = i_item_sk
-   and d_date in (select d_date
-  from date_dim
-  where d_week_seq = (select d_week_seq 
-  from date_dim
-  where d_date = '1998-02-19'))
-   and ss_sold_date_sk   = d_date_sk
- group by i_item_id),
- cs_items as
- (select i_item_id item_id
-,sum(cs_ext_sales_price) cs_item_rev
-  from catalog_sales
-  ,item
-  ,date_dim
- where cs_item_sk = i_item_sk
-  and  d_date in (select d_date
-  from date_dim
-  where d_week_seq = (select d_week_seq 
-  from date_dim
-  where d_date = '1998-02-19'))
-  and  cs_sold_date_sk = d_date_sk
- group by i_item_id),
- ws_items as
- (select i_item_id item_id
-,sum(ws_ext_sales_price) ws_item_rev
-  from web_sales
-  ,item
-  ,date_dim
- where ws_item_sk = i_item_sk
-  and  d_date in (select d_date
-  from date_dim
-  where d_week_seq =(select d_week_seq 
- from date_dim
- where d_date = '1998-02-19'))
-  and ws_sold_date_sk   = d_date_sk
- group by i_item_id)
-  select  ss_items.item_id
-   ,ss_item_rev
-   ,ss_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 ss_dev
-   ,cs_item_rev
-   ,cs_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 cs_dev
-   ,ws_item_rev
-   ,ws_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 ws_dev
-   ,(ss_item_rev+cs_item_rev+ws_item_rev)/3 average
- from ss_items,cs_items,ws_items
- where ss_items.item_id=cs_items.item_id
-   and ss_items.item_id=ws_items.item_id 
-   and ss_item_rev between 0.9 * cs_item_rev and 1.1 * cs_item_rev
-   and ss_item_rev between 0.9 * ws_item_rev and 1.1 * ws_item_rev
-   and cs_item_rev between 0.9 * ss_item_rev and 1.1 * ss_item_rev
-   and cs_item_rev between 0.9 * ws_item_rev and 1.1 * ws_item_rev
-   and ws_item_rev between 0.9 * ss_item_rev and 1.1 * ss_item_rev
-   and ws_item_rev between 0.9 * cs_item_rev and 1.1 * cs_item_rev
- order by item_id
- ,ss_item_rev
- limit 100
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-with ss_items as
- (select i_item_id item_id
-,sum(ss_ext_sales_price) ss_item_rev 
- from store_sales
- ,item
- ,date_dim
- where ss_item_sk = i_item_sk
-   and d_date in (select d_date
-  from date_dim
-  where d_week_seq = (select d_week_seq 
-  from date_dim
-  where d_date = '1998-02-19'))
-   and ss_sold_date_sk   = d_date_sk
- group by i_item_id),
- cs_items as
- (select i_item_id item_id
-,sum(cs_ext_sales_price) cs_item_rev
-  from catalog_sales
-  ,item
-  ,date_dim
- where cs_item_sk = i_item_sk
-  and  d_date in (select d_date
-  from date_dim
-  where d_week_seq = (select d_week_seq 
-  from date_dim
-  where d_date = '1998-02-19'))
-  and  cs_sold_date_sk = d_date_sk
- group by i_item_id),
- ws_items as
- (select i_item_id item_id
-,sum(ws_ext_sales_price) ws_item_rev
-  from web_sales
-  ,item
-  ,date_dim
- where ws_item_sk = i_item_sk
-  and  d_date in (select d_date
-  from date_dim
-  where d_week_seq =(select d_week_seq 
- from date_dim
- where d_date = '1998-02-19'))
-  and ws_sold_date_sk   = d_date_sk
- group by i_item_id)
-  select  ss_items.item_id
-   ,ss_item_rev
-   ,ss_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 ss_dev
-   ,cs_item_rev
-   ,cs_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 cs_dev
-   ,ws_item_rev
-   ,ws_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 ws_dev
-   ,(ss_item_rev+cs_item_rev+ws_item_

[31/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query54.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query54.q.out 
b/ql/src/test/results/clientpositive/perf/query54.q.out
deleted file mode 100644
index e6b9b28..000
--- a/ql/src/test/results/clientpositive/perf/query54.q.out
+++ /dev/null
@@ -1,386 +0,0 @@
-Warning: Shuffle Join MERGEJOIN[184][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3]] in Stage 'Reducer 4' is a cross product
-Warning: Shuffle Join MERGEJOIN[185][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4]] in Stage 'Reducer 5' is a cross product
-Warning: Shuffle Join MERGEJOIN[183][tables = [$hdt$_1, $hdt$_2]] in Stage 
'Reducer 30' is a cross product
-Warning: Shuffle Join MERGEJOIN[186][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in 
Stage 'Reducer 6' is a cross product
-PREHOOK: query: explain
-with my_customers as (
- select distinct c_customer_sk
-, c_current_addr_sk
- from   
-( select cs_sold_date_sk sold_date_sk,
- cs_bill_customer_sk customer_sk,
- cs_item_sk item_sk
-  from   catalog_sales
-  union all
-  select ws_sold_date_sk sold_date_sk,
- ws_bill_customer_sk customer_sk,
- ws_item_sk item_sk
-  from   web_sales
- ) cs_or_ws_sales,
- item,
- date_dim,
- customer
- where   sold_date_sk = d_date_sk
- and item_sk = i_item_sk
- and i_category = 'Jewelry'
- and i_class = 'consignment'
- and c_customer_sk = cs_or_ws_sales.customer_sk
- and d_moy = 3
- and d_year = 1999
- )
- , my_revenue as (
- select c_customer_sk,
-sum(ss_ext_sales_price) as revenue
- from   my_customers,
-store_sales,
-customer_address,
-store,
-date_dim
- where  c_current_addr_sk = ca_address_sk
-and ca_county = s_county
-and ca_state = s_state
-and ss_sold_date_sk = d_date_sk
-and c_customer_sk = ss_customer_sk
-and d_month_seq between (select distinct d_month_seq+1
- from   date_dim where d_year = 1999 and d_moy 
= 3)
-   and  (select distinct d_month_seq+3
- from   date_dim where d_year = 1999 and d_moy 
= 3)
- group by c_customer_sk
- )
- , segments as
- (select cast((revenue/50) as int) as segment
-  from   my_revenue
- )
-  select  segment, count(*) as num_customers, segment*50 as segment_base
- from segments
- group by segment
- order by segment, num_customers
- limit 100
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-with my_customers as (
- select distinct c_customer_sk
-, c_current_addr_sk
- from   
-( select cs_sold_date_sk sold_date_sk,
- cs_bill_customer_sk customer_sk,
- cs_item_sk item_sk
-  from   catalog_sales
-  union all
-  select ws_sold_date_sk sold_date_sk,
- ws_bill_customer_sk customer_sk,
- ws_item_sk item_sk
-  from   web_sales
- ) cs_or_ws_sales,
- item,
- date_dim,
- customer
- where   sold_date_sk = d_date_sk
- and item_sk = i_item_sk
- and i_category = 'Jewelry'
- and i_class = 'consignment'
- and c_customer_sk = cs_or_ws_sales.customer_sk
- and d_moy = 3
- and d_year = 1999
- )
- , my_revenue as (
- select c_customer_sk,
-sum(ss_ext_sales_price) as revenue
- from   my_customers,
-store_sales,
-customer_address,
-store,
-date_dim
- where  c_current_addr_sk = ca_address_sk
-and ca_county = s_county
-and ca_state = s_state
-and ss_sold_date_sk = d_date_sk
-and c_customer_sk = ss_customer_sk
-and d_month_seq between (select distinct d_month_seq+1
- from   date_dim where d_year = 1999 and d_moy 
= 3)
-   and  (select distinct d_month_seq+3
- from   date_dim where d_year = 1999 and d_moy 
= 3)
- group by c_customer_sk
- )
- , segments as
- (select cast((revenue/50) as int) as segment
-  from   my_revenue
- )
-  select  segment, count(*) as num_customers, segment*50 as segment_base
- from segments
- group by segment
- order by segment, num_customers
- limit 100
-POSTHOOK: type: QUERY
-Plan optimized by CBO.
-
-Vertex dependency in root stage
-Map 15 <- Union 16 (CONTAINS)
-Map 21 <- Union 16 (CONTAINS)
-Reducer 12 <- Map 11 (SIMPLE_EDGE), Map 14 (SIMPLE_EDGE)
-Reducer 13 <- Reducer 12 (SIMPLE_EDGE), Reducer 20 (SIMPLE_EDGE)
-Reducer 17 <- Map 22 (SIMPLE_EDGE), Union 16 (SIMPLE_EDGE)
-Reducer 18 <- Map 23 (SIMPLE_EDGE), Reducer 17 (SIMPLE_EDGE)
-Reducer 19 <- Map 24 (SIMPLE_EDGE), Reducer 18 (SIMPLE_EDGE)
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 10 (S

[15/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query33.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query33.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query33.q.out
new file mode 100644
index 000..0810c28
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query33.q.out
@@ -0,0 +1,393 @@
+PREHOOK: query: explain
+with ss as (
+ select
+  i_manufact_id,sum(ss_ext_sales_price) total_sales
+ from
+   store_sales,
+   date_dim,
+ customer_address,
+ item
+ where
+ i_manufact_id in (select
+  i_manufact_id
+from
+ item
+where i_category in ('Books'))
+ and ss_item_sk  = i_item_sk
+ and ss_sold_date_sk = d_date_sk
+ and d_year  = 1999
+ and d_moy   = 3
+ and ss_addr_sk  = ca_address_sk
+ and ca_gmt_offset   = -6 
+ group by i_manufact_id),
+ cs as (
+ select
+  i_manufact_id,sum(cs_ext_sales_price) total_sales
+ from
+   catalog_sales,
+   date_dim,
+ customer_address,
+ item
+ where
+ i_manufact_id   in (select
+  i_manufact_id
+from
+ item
+where i_category in ('Books'))
+ and cs_item_sk  = i_item_sk
+ and cs_sold_date_sk = d_date_sk
+ and d_year  = 1999
+ and d_moy   = 3
+ and cs_bill_addr_sk = ca_address_sk
+ and ca_gmt_offset   = -6 
+ group by i_manufact_id),
+ ws as (
+ select
+  i_manufact_id,sum(ws_ext_sales_price) total_sales
+ from
+   web_sales,
+   date_dim,
+ customer_address,
+ item
+ where
+ i_manufact_id   in (select
+  i_manufact_id
+from
+ item
+where i_category in ('Books'))
+ and ws_item_sk  = i_item_sk
+ and ws_sold_date_sk = d_date_sk
+ and d_year  = 1999
+ and d_moy   = 3
+ and ws_bill_addr_sk = ca_address_sk
+ and ca_gmt_offset   = -6
+ group by i_manufact_id)
+  select  i_manufact_id ,sum(total_sales) total_sales
+ from  (select * from ss 
+union all
+select * from cs 
+union all
+select * from ws) tmp1
+ group by i_manufact_id
+ order by total_sales
+limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with ss as (
+ select
+  i_manufact_id,sum(ss_ext_sales_price) total_sales
+ from
+   store_sales,
+   date_dim,
+ customer_address,
+ item
+ where
+ i_manufact_id in (select
+  i_manufact_id
+from
+ item
+where i_category in ('Books'))
+ and ss_item_sk  = i_item_sk
+ and ss_sold_date_sk = d_date_sk
+ and d_year  = 1999
+ and d_moy   = 3
+ and ss_addr_sk  = ca_address_sk
+ and ca_gmt_offset   = -6 
+ group by i_manufact_id),
+ cs as (
+ select
+  i_manufact_id,sum(cs_ext_sales_price) total_sales
+ from
+   catalog_sales,
+   date_dim,
+ customer_address,
+ item
+ where
+ i_manufact_id   in (select
+  i_manufact_id
+from
+ item
+where i_category in ('Books'))
+ and cs_item_sk  = i_item_sk
+ and cs_sold_date_sk = d_date_sk
+ and d_year  = 1999
+ and d_moy   = 3
+ and cs_bill_addr_sk = ca_address_sk
+ and ca_gmt_offset   = -6 
+ group by i_manufact_id),
+ ws as (
+ select
+  i_manufact_id,sum(ws_ext_sales_price) total_sales
+ from
+   web_sales,
+   date_dim,
+ customer_address,
+ item
+ where
+ i_manufact_id   in (select
+  i_manufact_id
+from
+ item
+where i_category in ('Books'))
+ and ws_item_sk  = i_item_sk
+ and ws_sold_date_sk = d_date_sk
+ and d_year  = 1999
+ and d_moy   = 3
+ and ws_bill_addr_sk = ca_address_sk
+ and ca_gmt_offset   = -6
+ group by i_manufact_id)
+  select  i_manufact_id ,sum(total_sales) total_sales
+ from  (select * from ss 
+union all
+select * from cs 
+union all
+select * from ws) tmp1
+ group by i_manufact_id
+ order by total_sales
+limit 100
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 10 <- Reducer 9 (SIMPLE_EDGE), Union 5 (CONTAINS)
+Reducer 11 <- Map 1 (SIMPLE_EDGE), Reducer 17 (ONE_TO_ONE_EDGE)
+Reducer 12 <- Reducer 11 (SIMPLE_EDGE), Reducer 25 (SIMPLE_EDGE)
+Reducer 13 <- Reducer 12 (SIMPLE_EDGE), Union 5 (CONTAINS)
+Reducer 15 <- Map 14 (SIMPLE_EDGE)
+Reducer 16 <- Map 14 (SIMPLE_EDGE)
+Reducer 17 <- Map 14 (SIMPLE_EDGE)
+Reducer 19 <- Map 18 (SIMPLE_EDGE), Map 21 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 15 (ONE_TO_ONE_EDG

[02/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query9.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query9.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query9.q.out
new file mode 100644
index 000..24de6f2
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query9.q.out
@@ -0,0 +1,414 @@
+Warning: Shuffle Join MERGEJOIN[171][tables = [$hdt$_0, $hdt$_1]] in Stage 
'Reducer 2' is a cross product
+Warning: Shuffle Join MERGEJOIN[172][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in 
Stage 'Reducer 3' is a cross product
+Warning: Shuffle Join MERGEJOIN[173][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3]] in Stage 'Reducer 4' is a cross product
+Warning: Shuffle Join MERGEJOIN[174][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4]] in Stage 'Reducer 5' is a cross product
+Warning: Shuffle Join MERGEJOIN[175][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5]] in Stage 'Reducer 6' is a cross product
+Warning: Shuffle Join MERGEJOIN[176][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6]] in Stage 'Reducer 7' is a cross product
+Warning: Shuffle Join MERGEJOIN[177][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7]] in Stage 'Reducer 8' is a cross 
product
+Warning: Shuffle Join MERGEJOIN[178][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8]] in Stage 'Reducer 9' is 
a cross product
+Warning: Shuffle Join MERGEJOIN[179][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8, $hdt$_9]] in Stage 
'Reducer 10' is a cross product
+Warning: Shuffle Join MERGEJOIN[180][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8, $hdt$_9, $hdt$_10]] in 
Stage 'Reducer 11' is a cross product
+Warning: Shuffle Join MERGEJOIN[181][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8, $hdt$_9, $hdt$_10, 
$hdt$_11]] in Stage 'Reducer 12' is a cross product
+Warning: Shuffle Join MERGEJOIN[182][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8, $hdt$_9, $hdt$_10, 
$hdt$_11, $hdt$_12]] in Stage 'Reducer 13' is a cross product
+Warning: Shuffle Join MERGEJOIN[183][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8, $hdt$_9, $hdt$_10, 
$hdt$_11, $hdt$_12, $hdt$_13]] in Stage 'Reducer 14' is a cross product
+Warning: Shuffle Join MERGEJOIN[184][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8, $hdt$_9, $hdt$_10, 
$hdt$_11, $hdt$_12, $hdt$_13, $hdt$_14]] in Stage 'Reducer 15' is a cross 
product
+Warning: Shuffle Join MERGEJOIN[185][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3, $hdt$_4, $hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8, $hdt$_9, $hdt$_10, 
$hdt$_11, $hdt$_12, $hdt$_13, $hdt$_14, $hdt$_15]] in Stage 'Reducer 16' is a 
cross product
+PREHOOK: query: explain
+select case when (select count(*) 
+  from store_sales 
+  where ss_quantity between 1 and 20) > 409437
+then (select avg(ss_ext_list_price) 
+  from store_sales 
+  where ss_quantity between 1 and 20) 
+else (select avg(ss_net_paid_inc_tax)
+  from store_sales
+  where ss_quantity between 1 and 20) end bucket1 ,
+   case when (select count(*)
+  from store_sales
+  where ss_quantity between 21 and 40) > 4595804
+then (select avg(ss_ext_list_price)
+  from store_sales
+  where ss_quantity between 21 and 40) 
+else (select avg(ss_net_paid_inc_tax)
+  from store_sales
+  where ss_quantity between 21 and 40) end bucket2,
+   case when (select count(*)
+  from store_sales
+  where ss_quantity between 41 and 60) > 7887297
+then (select avg(ss_ext_list_price)
+  from store_sales
+  where ss_quantity between 41 and 60)
+else (select avg(ss_net_paid_inc_tax)
+  from store_sales
+  where ss_quantity between 41 and 60) end bucket3,
+   case when (select count(*)
+  from store_sales
+  where ss_quantity between 61 and 80) > 10872978
+then (select avg(ss_ext_list_price)
+  from store_sales
+  where ss_quantity between 61 and 80)
+else (select avg(ss_net_paid_inc_tax)
+  from store_sales
+  where ss_quantity between 61 and 80) end bucket4,
+   case when (select count(*)
+  from store_sales
+  where ss_quantity between 81 and 100) > 43571537
+then (select avg(ss_ext

[12/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query5.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query5.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query5.q.out
new file mode 100644
index 000..ff197d2
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query5.q.out
@@ -0,0 +1,473 @@
+PREHOOK: query: explain
+with ssr as
+ (select s_store_id,
+sum(sales_price) as sales,
+sum(profit) as profit,
+sum(return_amt) as returns,
+sum(net_loss) as profit_loss
+ from
+  ( select  ss_store_sk as store_sk,
+ss_sold_date_sk  as date_sk,
+ss_ext_sales_price as sales_price,
+ss_net_profit as profit,
+cast(0 as decimal(7,2)) as return_amt,
+cast(0 as decimal(7,2)) as net_loss
+from store_sales
+union all
+select sr_store_sk as store_sk,
+   sr_returned_date_sk as date_sk,
+   cast(0 as decimal(7,2)) as sales_price,
+   cast(0 as decimal(7,2)) as profit,
+   sr_return_amt as return_amt,
+   sr_net_loss as net_loss
+from store_returns
+   ) salesreturns,
+ date_dim,
+ store
+ where date_sk = d_date_sk
+   and d_date between cast('1998-08-04' as date) 
+  and (cast('1998-08-04' as date) +  14 days)
+   and store_sk = s_store_sk
+ group by s_store_id)
+ ,
+ csr as
+ (select cp_catalog_page_id,
+sum(sales_price) as sales,
+sum(profit) as profit,
+sum(return_amt) as returns,
+sum(net_loss) as profit_loss
+ from
+  ( select  cs_catalog_page_sk as page_sk,
+cs_sold_date_sk  as date_sk,
+cs_ext_sales_price as sales_price,
+cs_net_profit as profit,
+cast(0 as decimal(7,2)) as return_amt,
+cast(0 as decimal(7,2)) as net_loss
+from catalog_sales
+union all
+select cr_catalog_page_sk as page_sk,
+   cr_returned_date_sk as date_sk,
+   cast(0 as decimal(7,2)) as sales_price,
+   cast(0 as decimal(7,2)) as profit,
+   cr_return_amount as return_amt,
+   cr_net_loss as net_loss
+from catalog_returns
+   ) salesreturns,
+ date_dim,
+ catalog_page
+ where date_sk = d_date_sk
+   and d_date between cast('1998-08-04' as date)
+  and (cast('1998-08-04' as date) +  14 days)
+   and page_sk = cp_catalog_page_sk
+ group by cp_catalog_page_id)
+ ,
+ wsr as
+ (select web_site_id,
+sum(sales_price) as sales,
+sum(profit) as profit,
+sum(return_amt) as returns,
+sum(net_loss) as profit_loss
+ from
+  ( select  ws_web_site_sk as wsr_web_site_sk,
+ws_sold_date_sk  as date_sk,
+ws_ext_sales_price as sales_price,
+ws_net_profit as profit,
+cast(0 as decimal(7,2)) as return_amt,
+cast(0 as decimal(7,2)) as net_loss
+from web_sales
+union all
+select ws_web_site_sk as wsr_web_site_sk,
+   wr_returned_date_sk as date_sk,
+   cast(0 as decimal(7,2)) as sales_price,
+   cast(0 as decimal(7,2)) as profit,
+   wr_return_amt as return_amt,
+   wr_net_loss as net_loss
+from web_returns left outer join web_sales on
+ ( wr_item_sk = ws_item_sk
+   and wr_order_number = ws_order_number)
+   ) salesreturns,
+ date_dim,
+ web_site
+ where date_sk = d_date_sk
+   and d_date between cast('1998-08-04' as date)
+  and (cast('1998-08-04' as date) +  14 days)
+   and wsr_web_site_sk = web_site_sk
+ group by web_site_id)
+  select  channel
+, id
+, sum(sales) as sales
+, sum(returns) as returns
+, sum(profit) as profit
+ from 
+ (select 'store channel' as channel
+, 'store' || s_store_id as id
+, sales
+, returns
+, (profit - profit_loss) as profit
+ from   ssr
+ union all
+ select 'catalog channel' as channel
+, 'catalog_page' || cp_catalog_page_id as id
+, sales
+, returns
+, (profit - profit_loss) as profit
+ from  csr
+ union all
+ select 'web channel' as channel
+, 'web_site' || web_site_id as id
+, sales
+, returns
+, (profit - profit_loss) as profit
+ from   wsr
+ ) x
+ group by rollup (channel, id)
+ order by channel
+ ,id
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with ssr as
+ (select s_store_id,
+sum(sales_price) as sales,
+sum(profit) as profit,
+sum(return_amt) as returns,
+sum(net_loss) as profit_loss
+ from
+  ( select  ss_store_sk as store_sk,
+ss_sold_date_sk  as date_sk,
+ss_ext_sales_price as sales_price,
+ss_net_profit as profit,
+cast(0 as decimal(7,2)) as return_amt,
+cast(0 as decimal(7,2)) as net

[04/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query81.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query81.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query81.q.out
new file mode 100644
index 000..5fb04b2
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query81.q.out
@@ -0,0 +1,214 @@
+PREHOOK: query: explain
+with customer_total_return as
+ (select cr_returning_customer_sk as ctr_customer_sk
+,ca_state as ctr_state, 
+   sum(cr_return_amt_inc_tax) as ctr_total_return
+ from catalog_returns
+ ,date_dim
+ ,customer_address
+ where cr_returned_date_sk = d_date_sk 
+   and d_year =1998
+   and cr_returning_addr_sk = ca_address_sk 
+ group by cr_returning_customer_sk
+ ,ca_state )
+  select  
c_customer_id,c_salutation,c_first_name,c_last_name,ca_street_number,ca_street_name
+   
,ca_street_type,ca_suite_number,ca_city,ca_county,ca_state,ca_zip,ca_country,ca_gmt_offset
+  ,ca_location_type,ctr_total_return
+ from customer_total_return ctr1
+ ,customer_address
+ ,customer
+ where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2
+ from customer_total_return ctr2 
+ where ctr1.ctr_state = ctr2.ctr_state)
+   and ca_address_sk = c_current_addr_sk
+   and ca_state = 'IL'
+   and ctr1.ctr_customer_sk = c_customer_sk
+ order by 
c_customer_id,c_salutation,c_first_name,c_last_name,ca_street_number,ca_street_name
+   
,ca_street_type,ca_suite_number,ca_city,ca_county,ca_state,ca_zip,ca_country,ca_gmt_offset
+  ,ca_location_type,ctr_total_return
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with customer_total_return as
+ (select cr_returning_customer_sk as ctr_customer_sk
+,ca_state as ctr_state, 
+   sum(cr_return_amt_inc_tax) as ctr_total_return
+ from catalog_returns
+ ,date_dim
+ ,customer_address
+ where cr_returned_date_sk = d_date_sk 
+   and d_year =1998
+   and cr_returning_addr_sk = ca_address_sk 
+ group by cr_returning_customer_sk
+ ,ca_state )
+  select  
c_customer_id,c_salutation,c_first_name,c_last_name,ca_street_number,ca_street_name
+   
,ca_street_type,ca_suite_number,ca_city,ca_county,ca_state,ca_zip,ca_country,ca_gmt_offset
+  ,ca_location_type,ctr_total_return
+ from customer_total_return ctr1
+ ,customer_address
+ ,customer
+ where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2
+ from customer_total_return ctr2 
+ where ctr1.ctr_state = ctr2.ctr_state)
+   and ca_address_sk = c_current_addr_sk
+   and ca_state = 'IL'
+   and ctr1.ctr_customer_sk = c_customer_sk
+ order by 
c_customer_id,c_salutation,c_first_name,c_last_name,ca_street_number,ca_street_name
+   
,ca_street_type,ca_suite_number,ca_city,ca_county,ca_state,ca_zip,ca_country,ca_gmt_offset
+  ,ca_location_type,ctr_total_return
+ limit 100
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 10 <- Reducer 13 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
+Reducer 11 <- Map 14 (SIMPLE_EDGE), Map 6 (SIMPLE_EDGE)
+Reducer 12 <- Map 15 (SIMPLE_EDGE), Reducer 11 (SIMPLE_EDGE)
+Reducer 13 <- Reducer 12 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 5 (SIMPLE_EDGE)
+Reducer 3 <- Reducer 10 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
+Reducer 7 <- Map 14 (SIMPLE_EDGE), Map 6 (SIMPLE_EDGE)
+Reducer 8 <- Map 15 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
+Reducer 9 <- Reducer 8 (SIMPLE_EDGE)
+
+Stage-0
+  Fetch Operator
+limit:-1
+Stage-1
+  Reducer 4
+  File Output Operator [FS_67]
+Select Operator [SEL_66] (rows=100 width=860)
+  
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13","_col14","_col15"]
+  Limit [LIM_65] (rows=100 width=860)
+Number of rows:100
+Select Operator [SEL_64] (rows=9683 width=860)
+  
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13","_col14"]
+<-Reducer 3 [SIMPLE_EDGE]
+  SHUFFLE [RS_63]
+Select Operator [SEL_62] (rows=9683 width=860)
+  
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13","_col14"]
+  Merge Join Operator [MERGEJOIN_108] (rows=9683 width=860)
+
Conds:RS_59._col0=RS_60._col0(Inner),Output:["_col1","_col3","_col4","_col5","_col7","_col8","_col9","_col10","_col11","_col12","_col14","_col15","_col16","_col17","_col20"]
+

[34/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query39.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query39.q.out 
b/ql/src/test/results/clientpositive/perf/query39.q.out
deleted file mode 100644
index 8e91494..000
--- a/ql/src/test/results/clientpositive/perf/query39.q.out
+++ /dev/null
@@ -1,255 +0,0 @@
-PREHOOK: query: explain
-with inv as
-(select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy
-   ,stdev,mean, case mean when 0 then null else stdev/mean end cov
- from(select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy
-,stddev_samp(inv_quantity_on_hand) stdev,avg(inv_quantity_on_hand) 
mean
-  from inventory
-  ,item
-  ,warehouse
-  ,date_dim
-  where inv_item_sk = i_item_sk
-and inv_warehouse_sk = w_warehouse_sk
-and inv_date_sk = d_date_sk
-and d_year =1999
-  group by w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy) foo
- where case mean when 0 then 0 else stdev/mean end > 1)
-select inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean, inv1.cov
-,inv2.w_warehouse_sk,inv2.i_item_sk,inv2.d_moy,inv2.mean, inv2.cov
-from inv inv1,inv inv2
-where inv1.i_item_sk = inv2.i_item_sk
-  and inv1.w_warehouse_sk =  inv2.w_warehouse_sk
-  and inv1.d_moy=4
-  and inv2.d_moy=4+1
-order by inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean,inv1.cov
-,inv2.d_moy,inv2.mean, inv2.cov
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-with inv as
-(select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy
-   ,stdev,mean, case mean when 0 then null else stdev/mean end cov
- from(select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy
-,stddev_samp(inv_quantity_on_hand) stdev,avg(inv_quantity_on_hand) 
mean
-  from inventory
-  ,item
-  ,warehouse
-  ,date_dim
-  where inv_item_sk = i_item_sk
-and inv_warehouse_sk = w_warehouse_sk
-and inv_date_sk = d_date_sk
-and d_year =1999
-  group by w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy) foo
- where case mean when 0 then 0 else stdev/mean end > 1)
-select inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean, inv1.cov
-,inv2.w_warehouse_sk,inv2.i_item_sk,inv2.d_moy,inv2.mean, inv2.cov
-from inv inv1,inv inv2
-where inv1.i_item_sk = inv2.i_item_sk
-  and inv1.w_warehouse_sk =  inv2.w_warehouse_sk
-  and inv1.d_moy=4
-  and inv2.d_moy=4+1
-order by inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean,inv1.cov
-,inv2.d_moy,inv2.mean, inv2.cov
-POSTHOOK: type: QUERY
-Plan optimized by CBO.
-
-Vertex dependency in root stage
-Reducer 10 <- Map 14 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
-Reducer 11 <- Reducer 10 (SIMPLE_EDGE)
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 12 (SIMPLE_EDGE)
-Reducer 3 <- Map 13 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Map 14 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
-Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
-Reducer 6 <- Reducer 11 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
-Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
-Reducer 8 <- Map 1 (SIMPLE_EDGE), Map 12 (SIMPLE_EDGE)
-Reducer 9 <- Map 13 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
-
-Stage-0
-  Fetch Operator
-limit:-1
-Stage-1
-  Reducer 7
-  File Output Operator [FS_63]
-Select Operator [SEL_61] (rows=13756683 width=15)
-  
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9"]
-<-Reducer 6 [SIMPLE_EDGE]
-  SHUFFLE [RS_60]
-Select Operator [SEL_59] (rows=13756683 width=15)
-  
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7"]
-  Merge Join Operator [MERGEJOIN_104] (rows=13756683 width=15)
-Conds:RS_56._col1, _col2=RS_57._col1, 
_col2(Inner),Output:["_col1","_col2","_col3","_col4","_col6","_col7","_col8","_col9"]
-  <-Reducer 11 [SIMPLE_EDGE]
-SHUFFLE [RS_57]
-  PartitionCols:_col1, _col2
-  Select Operator [SEL_55] (rows=12506076 width=15)
-Output:["_col1","_col2","_col3","_col4"]
-Filter Operator [FIL_54] (rows=12506076 width=15)
-  predicate:CASE WHEN ((_col4 = 0.0)) THEN (false) ELSE 
(((_col3 / _col4) > 1.0)) END
-  Select Operator [SEL_53] (rows=25012152 width=15)
-Output:["_col1","_col2","_col3","_col4"]
-Group By Operator [GBY_52] (rows=25012152 width=15)
-  
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["stddev_samp(VALUE._col0)","avg(VALUE._col1)"],keys:KEY._col0,
 KEY._col1, KEY._col2
-<-Reducer 10 [SIMPLE_EDGE]
-  SHUFFLE [RS_51]
-PartitionCols:_col0, _col1, _col2
-Group By Operator [GB

[16/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query27.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query27.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query27.q.out
new file mode 100644
index 000..c6a1905
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query27.q.out
@@ -0,0 +1,140 @@
+PREHOOK: query: explain
+select  i_item_id,
+s_state, grouping(s_state) g_state,
+avg(ss_quantity) agg1,
+avg(ss_list_price) agg2,
+avg(ss_coupon_amt) agg3,
+avg(ss_sales_price) agg4
+ from store_sales, customer_demographics, date_dim, store, item
+ where ss_sold_date_sk = d_date_sk and
+   ss_item_sk = i_item_sk and
+   ss_store_sk = s_store_sk and
+   ss_cdemo_sk = cd_demo_sk and
+   cd_gender = 'M' and
+   cd_marital_status = 'U' and
+   cd_education_status = '2 yr Degree' and
+   d_year = 2001 and
+   s_state in ('SD','FL', 'MI', 'LA', 'MO', 'SC')
+ group by rollup (i_item_id, s_state)
+ order by i_item_id
+ ,s_state
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select  i_item_id,
+s_state, grouping(s_state) g_state,
+avg(ss_quantity) agg1,
+avg(ss_list_price) agg2,
+avg(ss_coupon_amt) agg3,
+avg(ss_sales_price) agg4
+ from store_sales, customer_demographics, date_dim, store, item
+ where ss_sold_date_sk = d_date_sk and
+   ss_item_sk = i_item_sk and
+   ss_store_sk = s_store_sk and
+   ss_cdemo_sk = cd_demo_sk and
+   cd_gender = 'M' and
+   cd_marital_status = 'U' and
+   cd_education_status = '2 yr Degree' and
+   d_year = 2001 and
+   s_state in ('SD','FL', 'MI', 'LA', 'MO', 'SC')
+ group by rollup (i_item_id, s_state)
+ order by i_item_id
+ ,s_state
+ limit 100
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 8 (SIMPLE_EDGE)
+Reducer 3 <- Map 9 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Map 10 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
+Reducer 5 <- Map 11 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
+Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
+Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
+
+Stage-0
+  Fetch Operator
+limit:100
+Stage-1
+  Reducer 7
+  File Output Operator [FS_36]
+Limit [LIM_35] (rows=100 width=88)
+  Number of rows:100
+  Select Operator [SEL_34] (rows=1264972921 width=88)
+Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"]
+  <-Reducer 6 [SIMPLE_EDGE]
+SHUFFLE [RS_33]
+  Select Operator [SEL_32] (rows=1264972921 width=88)
+
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"]
+Group By Operator [GBY_31] (rows=1264972921 width=88)
+  
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"],aggregations:["avg(VALUE._col0)","avg(VALUE._col1)","avg(VALUE._col2)","avg(VALUE._col3)"],keys:KEY._col0,
 KEY._col1, KEY._col2
+<-Reducer 5 [SIMPLE_EDGE]
+  SHUFFLE [RS_30]
+PartitionCols:_col0, _col1, _col2
+Group By Operator [GBY_29] (rows=2529945843 width=88)
+  
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"],aggregations:["avg(_col2)","avg(_col3)","avg(_col4)","avg(_col5)"],keys:_col0,
 _col1, 0
+  Select Operator [SEL_27] (rows=843315281 width=88)
+
Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
+Merge Join Operator [MERGEJOIN_59] (rows=843315281 
width=88)
+  
Conds:RS_24._col1=RS_25._col0(Inner),Output:["_col4","_col5","_col6","_col7","_col15","_col17"]
+<-Map 11 [SIMPLE_EDGE]
+  SHUFFLE [RS_25]
+PartitionCols:_col0
+Select Operator [SEL_14] (rows=462000 width=1436)
+  Output:["_col0","_col1"]
+  Filter Operator [FIL_55] (rows=462000 width=1436)
+predicate:i_item_sk is not null
+TableScan [TS_12] (rows=462000 width=1436)
+  
default@item,item,Tbl:COMPLETE,Col:NONE,Output:["i_item_sk","i_item_id"]
+<-Reducer 4 [SIMPLE_EDGE]
+  SHUFFLE [RS_24]
+PartitionCols:_col1
+Merge Join Operator [MERGEJOIN_58] (rows=766650239 
width=88)
+  
Conds:RS_21._col3=RS_22._col0(Inner),Output:["_col1","_col4","_col5","_col6","_col7","_col15"]
+<-Map 10 [SIMPLE_EDGE]
+  SHUFFLE [RS_22]
+Pa

[08/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query65.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query65.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query65.q.out
new file mode 100644
index 000..0091ad0
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query65.q.out
@@ -0,0 +1,169 @@
+PREHOOK: query: explain
+select 
+   s_store_name,
+   i_item_desc,
+   sc.revenue,
+   i_current_price,
+   i_wholesale_cost,
+   i_brand
+ from store, item,
+ (select ss_store_sk, avg(revenue) as ave
+   from
+   (select  ss_store_sk, ss_item_sk, 
+sum(ss_sales_price) as revenue
+   from store_sales, date_dim
+   where ss_sold_date_sk = d_date_sk and d_month_seq between 1212 
and 1212+11
+   group by ss_store_sk, ss_item_sk) sa
+   group by ss_store_sk) sb,
+ (select  ss_store_sk, ss_item_sk, sum(ss_sales_price) as revenue
+   from store_sales, date_dim
+   where ss_sold_date_sk = d_date_sk and d_month_seq between 1212 and 
1212+11
+   group by ss_store_sk, ss_item_sk) sc
+ where sb.ss_store_sk = sc.ss_store_sk and 
+   sc.revenue <= 0.1 * sb.ave and
+   s_store_sk = sc.ss_store_sk and
+   i_item_sk = sc.ss_item_sk
+ order by s_store_name, i_item_desc
+limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select 
+   s_store_name,
+   i_item_desc,
+   sc.revenue,
+   i_current_price,
+   i_wholesale_cost,
+   i_brand
+ from store, item,
+ (select ss_store_sk, avg(revenue) as ave
+   from
+   (select  ss_store_sk, ss_item_sk, 
+sum(ss_sales_price) as revenue
+   from store_sales, date_dim
+   where ss_sold_date_sk = d_date_sk and d_month_seq between 1212 
and 1212+11
+   group by ss_store_sk, ss_item_sk) sa
+   group by ss_store_sk) sb,
+ (select  ss_store_sk, ss_item_sk, sum(ss_sales_price) as revenue
+   from store_sales, date_dim
+   where ss_sold_date_sk = d_date_sk and d_month_seq between 1212 and 
1212+11
+   group by ss_store_sk, ss_item_sk) sc
+ where sb.ss_store_sk = sc.ss_store_sk and 
+   sc.revenue <= 0.1 * sb.ave and
+   s_store_sk = sc.ss_store_sk and
+   i_item_sk = sc.ss_item_sk
+ order by s_store_name, i_item_desc
+limit 100
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 9 (SIMPLE_EDGE)
+Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Map 10 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE), Reducer 8 
(SIMPLE_EDGE)
+Reducer 5 <- Map 11 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
+Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
+Reducer 7 <- Map 1 (SIMPLE_EDGE), Map 9 (SIMPLE_EDGE)
+Reducer 8 <- Reducer 7 (SIMPLE_EDGE)
+
+Stage-0
+  Fetch Operator
+limit:100
+Stage-1
+  Reducer 6
+  File Output Operator [FS_51]
+Limit [LIM_50] (rows=100 width=88)
+  Number of rows:100
+  Select Operator [SEL_49] (rows=20079 width=88)
+Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
+  <-Reducer 5 [SIMPLE_EDGE]
+SHUFFLE [RS_48]
+  Select Operator [SEL_47] (rows=20079 width=88)
+Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
+Merge Join Operator [MERGEJOIN_81] (rows=20079 width=88)
+  
Conds:RS_44._col1=RS_45._col0(Inner),Output:["_col2","_col6","_col8","_col9","_col10","_col11"]
+<-Map 11 [SIMPLE_EDGE]
+  SHUFFLE [RS_45]
+PartitionCols:_col0
+Select Operator [SEL_38] (rows=462000 width=1436)
+  Output:["_col0","_col1","_col2","_col3","_col4"]
+  Filter Operator [FIL_77] (rows=462000 width=1436)
+predicate:i_item_sk is not null
+TableScan [TS_36] (rows=462000 width=1436)
+  
default@item,item,Tbl:COMPLETE,Col:NONE,Output:["i_item_sk","i_item_desc","i_current_price","i_wholesale_cost","i_brand"]
+<-Reducer 4 [SIMPLE_EDGE]
+  SHUFFLE [RS_44]
+PartitionCols:_col1
+Filter Operator [FIL_43] (rows=232318249 width=88)
+  predicate:(_col2 <= (0.1 * _col4))
+  Merge Join Operator [MERGEJOIN_80] (rows=696954748 
width=88)
+
Conds:RS_39._col0=RS_40._col0(Inner),RS_39._col0=RS_41._col0(Inner),Output:["_col1","_col2","_col4","_col6"]
+  <-Map 10 [SIMPLE_EDGE]
+SHUFFLE [RS_41]
+  PartitionCols:_col0
+  Select Operator [SEL_35] (rows=1704 width=1910)
+Output:["_col0","_col1"

[20/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query11.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query11.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query11.q.out
new file mode 100644
index 000..69ced26
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query11.q.out
@@ -0,0 +1,351 @@
+PREHOOK: query: explain
+with year_total as (
+ select c_customer_id customer_id
+   ,c_first_name customer_first_name
+   ,c_last_name customer_last_name
+   ,c_preferred_cust_flag
+   ,c_birth_country customer_birth_country
+   ,c_login customer_login
+   ,c_email_address customer_email_address
+   ,d_year dyear
+   ,sum(ss_ext_list_price-ss_ext_discount_amt) year_total
+   ,'s' sale_type
+ from customer
+ ,store_sales
+ ,date_dim
+ where c_customer_sk = ss_customer_sk
+   and ss_sold_date_sk = d_date_sk
+ group by c_customer_id
+ ,c_first_name
+ ,c_last_name
+ ,d_year
+ ,c_preferred_cust_flag
+ ,c_birth_country
+ ,c_login
+ ,c_email_address
+ ,d_year 
+ union all
+ select c_customer_id customer_id
+   ,c_first_name customer_first_name
+   ,c_last_name customer_last_name
+   ,c_preferred_cust_flag
+   ,c_birth_country customer_birth_country
+   ,c_login customer_login
+   ,c_email_address customer_email_address
+   ,d_year dyear
+   ,sum(ws_ext_list_price-ws_ext_discount_amt) year_total
+   ,'w' sale_type
+ from customer
+ ,web_sales
+ ,date_dim
+ where c_customer_sk = ws_bill_customer_sk
+   and ws_sold_date_sk = d_date_sk
+ group by c_customer_id
+ ,c_first_name
+ ,c_last_name
+ ,c_preferred_cust_flag
+ ,c_birth_country
+ ,c_login
+ ,c_email_address
+ ,d_year
+ )
+  select  t_s_secyear.c_preferred_cust_flag
+ from year_total t_s_firstyear
+ ,year_total t_s_secyear
+ ,year_total t_w_firstyear
+ ,year_total t_w_secyear
+ where t_s_secyear.customer_id = t_s_firstyear.customer_id
+ and t_s_firstyear.customer_id = t_w_secyear.customer_id
+ and t_s_firstyear.customer_id = t_w_firstyear.customer_id
+ and t_s_firstyear.sale_type = 's'
+ and t_w_firstyear.sale_type = 'w'
+ and t_s_secyear.sale_type = 's'
+ and t_w_secyear.sale_type = 'w'
+ and t_s_firstyear.dyear = 2001
+ and t_s_secyear.dyear = 2001+1
+ and t_w_firstyear.dyear = 2001
+ and t_w_secyear.dyear = 2001+1
+ and t_s_firstyear.year_total > 0
+ and t_w_firstyear.year_total > 0
+ and case when t_w_firstyear.year_total > 0 then 
t_w_secyear.year_total / t_w_firstyear.year_total else null end
+ > case when t_s_firstyear.year_total > 0 then 
t_s_secyear.year_total / t_s_firstyear.year_total else null end
+ order by t_s_secyear.c_preferred_cust_flag
+limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with year_total as (
+ select c_customer_id customer_id
+   ,c_first_name customer_first_name
+   ,c_last_name customer_last_name
+   ,c_preferred_cust_flag
+   ,c_birth_country customer_birth_country
+   ,c_login customer_login
+   ,c_email_address customer_email_address
+   ,d_year dyear
+   ,sum(ss_ext_list_price-ss_ext_discount_amt) year_total
+   ,'s' sale_type
+ from customer
+ ,store_sales
+ ,date_dim
+ where c_customer_sk = ss_customer_sk
+   and ss_sold_date_sk = d_date_sk
+ group by c_customer_id
+ ,c_first_name
+ ,c_last_name
+ ,d_year
+ ,c_preferred_cust_flag
+ ,c_birth_country
+ ,c_login
+ ,c_email_address
+ ,d_year 
+ union all
+ select c_customer_id customer_id
+   ,c_first_name customer_first_name
+   ,c_last_name customer_last_name
+   ,c_preferred_cust_flag
+   ,c_birth_country customer_birth_country
+   ,c_login customer_login
+   ,c_email_address customer_email_address
+   ,d_year dyear
+   ,sum(ws_ext_list_price-ws_ext_discount_amt) year_total
+   ,'w' sale_type
+ from customer
+ ,web_sales
+ ,date_dim
+ where c_customer_sk = ws_bill_customer_sk
+   and ws_sold_date_sk = d_date_sk
+ group by c_customer_id
+ ,c_first_name
+ ,c_last_name
+ ,c_preferred_cust_flag
+ ,c_birth_country
+ ,c_login
+ ,c_email_address
+ ,d_year
+ )
+  select  t_s_secyear.c_preferred_cust_flag
+ from year_total t_s_firstyear
+ ,year_total t_s_secyear
+ ,year_total t_w_firstyear
+ ,year_total t_w_secyear
+ where t_s_secyear.customer_id = t_s_firstyear.customer_id
+ and t_s_firstyear.customer_id = t_w_secyear.customer_id
+ and t_s_firstyear.customer_id = t_w_firstyear.customer_id
+ and t_s_firstyear.sale_type = 's'
+ a

[29/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query61.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query61.q.out 
b/ql/src/test/results/clientpositive/perf/query61.q.out
deleted file mode 100644
index 0b4f5fd..000
--- a/ql/src/test/results/clientpositive/perf/query61.q.out
+++ /dev/null
@@ -1,280 +0,0 @@
-Warning: Shuffle Join MERGEJOIN[156][tables = [$hdt$_0, $hdt$_1]] in Stage 
'Reducer 5' is a cross product
-PREHOOK: query: explain
-select  promotions,total,cast(promotions as decimal(15,4))/cast(total as 
decimal(15,4))*100
-from
-  (select sum(ss_ext_sales_price) promotions
-   from  store_sales
-,store
-,promotion
-,date_dim
-,customer
-,customer_address 
-,item
-   where ss_sold_date_sk = d_date_sk
-   and   ss_store_sk = s_store_sk
-   and   ss_promo_sk = p_promo_sk
-   and   ss_customer_sk= c_customer_sk
-   and   ca_address_sk = c_current_addr_sk
-   and   ss_item_sk = i_item_sk 
-   and   ca_gmt_offset = -7
-   and   i_category = 'Electronics'
-   and   (p_channel_dmail = 'Y' or p_channel_email = 'Y' or p_channel_tv = 'Y')
-   and   s_gmt_offset = -7
-   and   d_year = 1999
-   and   d_moy  = 11) promotional_sales,
-  (select sum(ss_ext_sales_price) total
-   from  store_sales
-,store
-,date_dim
-,customer
-,customer_address
-,item
-   where ss_sold_date_sk = d_date_sk
-   and   ss_store_sk = s_store_sk
-   and   ss_customer_sk= c_customer_sk
-   and   ca_address_sk = c_current_addr_sk
-   and   ss_item_sk = i_item_sk
-   and   ca_gmt_offset = -7
-   and   i_category = 'Electronics'
-   and   s_gmt_offset = -7
-   and   d_year = 1999
-   and   d_moy  = 11) all_sales
-order by promotions, total
-limit 100
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-select  promotions,total,cast(promotions as decimal(15,4))/cast(total as 
decimal(15,4))*100
-from
-  (select sum(ss_ext_sales_price) promotions
-   from  store_sales
-,store
-,promotion
-,date_dim
-,customer
-,customer_address 
-,item
-   where ss_sold_date_sk = d_date_sk
-   and   ss_store_sk = s_store_sk
-   and   ss_promo_sk = p_promo_sk
-   and   ss_customer_sk= c_customer_sk
-   and   ca_address_sk = c_current_addr_sk
-   and   ss_item_sk = i_item_sk 
-   and   ca_gmt_offset = -7
-   and   i_category = 'Electronics'
-   and   (p_channel_dmail = 'Y' or p_channel_email = 'Y' or p_channel_tv = 'Y')
-   and   s_gmt_offset = -7
-   and   d_year = 1999
-   and   d_moy  = 11) promotional_sales,
-  (select sum(ss_ext_sales_price) total
-   from  store_sales
-,store
-,date_dim
-,customer
-,customer_address
-,item
-   where ss_sold_date_sk = d_date_sk
-   and   ss_store_sk = s_store_sk
-   and   ss_customer_sk= c_customer_sk
-   and   ca_address_sk = c_current_addr_sk
-   and   ss_item_sk = i_item_sk
-   and   ca_gmt_offset = -7
-   and   i_category = 'Electronics'
-   and   s_gmt_offset = -7
-   and   d_year = 1999
-   and   d_moy  = 11) all_sales
-order by promotions, total
-limit 100
-POSTHOOK: type: QUERY
-Plan optimized by CBO.
-
-Vertex dependency in root stage
-Reducer 12 <- Map 11 (SIMPLE_EDGE), Map 16 (SIMPLE_EDGE)
-Reducer 13 <- Map 20 (SIMPLE_EDGE), Reducer 12 (SIMPLE_EDGE)
-Reducer 14 <- Map 21 (SIMPLE_EDGE), Reducer 13 (SIMPLE_EDGE)
-Reducer 15 <- Map 22 (SIMPLE_EDGE), Reducer 14 (SIMPLE_EDGE)
-Reducer 17 <- Map 16 (SIMPLE_EDGE), Map 23 (SIMPLE_EDGE)
-Reducer 18 <- Map 20 (SIMPLE_EDGE), Reducer 17 (SIMPLE_EDGE)
-Reducer 19 <- Map 21 (SIMPLE_EDGE), Reducer 18 (SIMPLE_EDGE)
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 10 (SIMPLE_EDGE)
-Reducer 3 <- Reducer 15 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Reducer 3 (CUSTOM_SIMPLE_EDGE)
-Reducer 5 <- Reducer 4 (CUSTOM_SIMPLE_EDGE), Reducer 9 (CUSTOM_SIMPLE_EDGE)
-Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
-Reducer 7 <- Map 1 (SIMPLE_EDGE), Map 10 (SIMPLE_EDGE)
-Reducer 8 <- Reducer 19 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
-Reducer 9 <- Reducer 8 (CUSTOM_SIMPLE_EDGE)
-
-Stage-0
-  Fetch Operator
-limit:100
-Stage-1
-  Reducer 6
-  File Output Operator [FS_91]
-Limit [LIM_90] (rows=1 width=225)
-  Number of rows:100
-  Select Operator [SEL_89] (rows=1 width=225)
-Output:["_col0","_col1","_col2"]
-  <-Reducer 5 [SIMPLE_EDGE]
-SHUFFLE [RS_88]
-  Select Operator [SEL_87] (rows=1 width=225)
-Output:["_col0","_col1","_col2"]
-Merge Join Operator [MERGEJOIN_156] (rows=1 width=225)
-  Conds:(Inner),Output:["_col0","_col1"]
-<-Reducer 4 [CUSTOM_SIMPLE_EDGE]
-  PARTITION_ONLY_SHUFFLE [RS_84]
-Group By Operator [GBY_43] (rows=1 width=112)
-  Output:["_col0"],aggregations:["sum(VALUE._col0)"]
-

[10/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query58.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query58.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query58.q.out
new file mode 100644
index 000..d5ef23b
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query58.q.out
@@ -0,0 +1,440 @@
+Warning: Shuffle Join MERGEJOIN[265][tables = [$hdt$_4, $hdt$_5]] in Stage 
'Reducer 20' is a cross product
+Warning: Shuffle Join MERGEJOIN[269][tables = [$hdt$_5, $hdt$_6]] in Stage 
'Reducer 24' is a cross product
+Warning: Shuffle Join MERGEJOIN[273][tables = [$hdt$_6, $hdt$_7]] in Stage 
'Reducer 28' is a cross product
+PREHOOK: query: explain
+with ss_items as
+ (select i_item_id item_id
+,sum(ss_ext_sales_price) ss_item_rev 
+ from store_sales
+ ,item
+ ,date_dim
+ where ss_item_sk = i_item_sk
+   and d_date in (select d_date
+  from date_dim
+  where d_week_seq = (select d_week_seq 
+  from date_dim
+  where d_date = '1998-02-19'))
+   and ss_sold_date_sk   = d_date_sk
+ group by i_item_id),
+ cs_items as
+ (select i_item_id item_id
+,sum(cs_ext_sales_price) cs_item_rev
+  from catalog_sales
+  ,item
+  ,date_dim
+ where cs_item_sk = i_item_sk
+  and  d_date in (select d_date
+  from date_dim
+  where d_week_seq = (select d_week_seq 
+  from date_dim
+  where d_date = '1998-02-19'))
+  and  cs_sold_date_sk = d_date_sk
+ group by i_item_id),
+ ws_items as
+ (select i_item_id item_id
+,sum(ws_ext_sales_price) ws_item_rev
+  from web_sales
+  ,item
+  ,date_dim
+ where ws_item_sk = i_item_sk
+  and  d_date in (select d_date
+  from date_dim
+  where d_week_seq =(select d_week_seq 
+ from date_dim
+ where d_date = '1998-02-19'))
+  and ws_sold_date_sk   = d_date_sk
+ group by i_item_id)
+  select  ss_items.item_id
+   ,ss_item_rev
+   ,ss_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 ss_dev
+   ,cs_item_rev
+   ,cs_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 cs_dev
+   ,ws_item_rev
+   ,ws_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 ws_dev
+   ,(ss_item_rev+cs_item_rev+ws_item_rev)/3 average
+ from ss_items,cs_items,ws_items
+ where ss_items.item_id=cs_items.item_id
+   and ss_items.item_id=ws_items.item_id 
+   and ss_item_rev between 0.9 * cs_item_rev and 1.1 * cs_item_rev
+   and ss_item_rev between 0.9 * ws_item_rev and 1.1 * ws_item_rev
+   and cs_item_rev between 0.9 * ss_item_rev and 1.1 * ss_item_rev
+   and cs_item_rev between 0.9 * ws_item_rev and 1.1 * ws_item_rev
+   and ws_item_rev between 0.9 * ss_item_rev and 1.1 * ss_item_rev
+   and ws_item_rev between 0.9 * cs_item_rev and 1.1 * cs_item_rev
+ order by item_id
+ ,ss_item_rev
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with ss_items as
+ (select i_item_id item_id
+,sum(ss_ext_sales_price) ss_item_rev 
+ from store_sales
+ ,item
+ ,date_dim
+ where ss_item_sk = i_item_sk
+   and d_date in (select d_date
+  from date_dim
+  where d_week_seq = (select d_week_seq 
+  from date_dim
+  where d_date = '1998-02-19'))
+   and ss_sold_date_sk   = d_date_sk
+ group by i_item_id),
+ cs_items as
+ (select i_item_id item_id
+,sum(cs_ext_sales_price) cs_item_rev
+  from catalog_sales
+  ,item
+  ,date_dim
+ where cs_item_sk = i_item_sk
+  and  d_date in (select d_date
+  from date_dim
+  where d_week_seq = (select d_week_seq 
+  from date_dim
+  where d_date = '1998-02-19'))
+  and  cs_sold_date_sk = d_date_sk
+ group by i_item_id),
+ ws_items as
+ (select i_item_id item_id
+,sum(ws_ext_sales_price) ws_item_rev
+  from web_sales
+  ,item
+  ,date_dim
+ where ws_item_sk = i_item_sk
+  and  d_date in (select d_date
+  from date_dim
+  where d_week_seq =(select d_week_seq 
+ from date_dim
+ where d_date = '1998-02-19'))
+  and ws_sold_date_sk   = d_date_sk
+ group by i_item_id)
+  select  ss_items.item_id
+   ,ss_item_rev
+   ,ss_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 ss_dev
+   ,cs_item_rev
+   ,cs_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 cs_dev
+   ,ws_item_rev
+   ,ws_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 ws_dev
+   ,(ss_item_rev+cs_item_

[07/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query7.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query7.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query7.q.out
new file mode 100644
index 000..00628db
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query7.q.out
@@ -0,0 +1,132 @@
+PREHOOK: query: explain
+select  i_item_id, 
+avg(ss_quantity) agg1,
+avg(ss_list_price) agg2,
+avg(ss_coupon_amt) agg3,
+avg(ss_sales_price) agg4 
+ from store_sales, customer_demographics, date_dim, item, promotion
+ where ss_sold_date_sk = d_date_sk and
+   ss_item_sk = i_item_sk and
+   ss_cdemo_sk = cd_demo_sk and
+   ss_promo_sk = p_promo_sk and
+   cd_gender = 'F' and 
+   cd_marital_status = 'W' and
+   cd_education_status = 'Primary' and
+   (p_channel_email = 'N' or p_channel_event = 'N') and
+   d_year = 1998 
+ group by i_item_id
+ order by i_item_id
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select  i_item_id, 
+avg(ss_quantity) agg1,
+avg(ss_list_price) agg2,
+avg(ss_coupon_amt) agg3,
+avg(ss_sales_price) agg4 
+ from store_sales, customer_demographics, date_dim, item, promotion
+ where ss_sold_date_sk = d_date_sk and
+   ss_item_sk = i_item_sk and
+   ss_cdemo_sk = cd_demo_sk and
+   ss_promo_sk = p_promo_sk and
+   cd_gender = 'F' and 
+   cd_marital_status = 'W' and
+   cd_education_status = 'Primary' and
+   (p_channel_email = 'N' or p_channel_event = 'N') and
+   d_year = 1998 
+ group by i_item_id
+ order by i_item_id
+ limit 100
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 8 (SIMPLE_EDGE)
+Reducer 3 <- Map 9 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Map 10 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
+Reducer 5 <- Map 11 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
+Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
+Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
+
+Stage-0
+  Fetch Operator
+limit:100
+Stage-1
+  Reducer 7
+  File Output Operator [FS_35]
+Limit [LIM_34] (rows=100 width=88)
+  Number of rows:100
+  Select Operator [SEL_33] (rows=421657640 width=88)
+Output:["_col0","_col1","_col2","_col3","_col4"]
+  <-Reducer 6 [SIMPLE_EDGE]
+SHUFFLE [RS_32]
+  Group By Operator [GBY_30] (rows=421657640 width=88)
+
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["avg(VALUE._col0)","avg(VALUE._col1)","avg(VALUE._col2)","avg(VALUE._col3)"],keys:KEY._col0
+  <-Reducer 5 [SIMPLE_EDGE]
+SHUFFLE [RS_29]
+  PartitionCols:_col0
+  Group By Operator [GBY_28] (rows=843315281 width=88)
+
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["avg(_col4)","avg(_col5)","avg(_col7)","avg(_col6)"],keys:_col18
+Merge Join Operator [MERGEJOIN_58] (rows=843315281 
width=88)
+  
Conds:RS_24._col1=RS_25._col0(Inner),Output:["_col4","_col5","_col6","_col7","_col18"]
+<-Map 11 [SIMPLE_EDGE]
+  SHUFFLE [RS_25]
+PartitionCols:_col0
+Select Operator [SEL_14] (rows=462000 width=1436)
+  Output:["_col0","_col1"]
+  Filter Operator [FIL_54] (rows=462000 width=1436)
+predicate:i_item_sk is not null
+TableScan [TS_12] (rows=462000 width=1436)
+  
default@item,item,Tbl:COMPLETE,Col:NONE,Output:["i_item_sk","i_item_id"]
+<-Reducer 4 [SIMPLE_EDGE]
+  SHUFFLE [RS_24]
+PartitionCols:_col1
+Merge Join Operator [MERGEJOIN_57] (rows=766650239 
width=88)
+  
Conds:RS_21._col3=RS_22._col0(Inner),Output:["_col1","_col4","_col5","_col6","_col7"]
+<-Map 10 [SIMPLE_EDGE]
+  SHUFFLE [RS_22]
+PartitionCols:_col0
+Select Operator [SEL_11] (rows=2300 width=1179)
+  Output:["_col0"]
+  Filter Operator [FIL_53] (rows=2300 width=1179)
+predicate:(((p_channel_email = 'N') or 
(p_channel_event = 'N')) and p_promo_sk is not null)
+TableScan [TS_9] (rows=2300 width=1179)
+  
default@promotion,promotion,Tbl:COMPLETE,Col:NONE,Output:["p_promo_sk","p_channel_email","p_channel_event"]
+<-Reducer 3 [SIMPLE_EDGE]
+  SHUFFLE [RS_21]
+

[38/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query15.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query15.q.out 
b/ql/src/test/results/clientpositive/perf/query15.q.out
deleted file mode 100644
index cf6cf16..000
--- a/ql/src/test/results/clientpositive/perf/query15.q.out
+++ /dev/null
@@ -1,119 +0,0 @@
-PREHOOK: query: explain
-select  ca_zip
-   ,sum(cs_sales_price)
- from catalog_sales
- ,customer
- ,customer_address
- ,date_dim
- where cs_bill_customer_sk = c_customer_sk
-   and c_current_addr_sk = ca_address_sk 
-   and ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475',
-   '85392', '85460', '80348', '81792')
- or ca_state in ('CA','WA','GA')
- or cs_sales_price > 500)
-   and cs_sold_date_sk = d_date_sk
-   and d_qoy = 2 and d_year = 2000
- group by ca_zip
- order by ca_zip
- limit 100
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-select  ca_zip
-   ,sum(cs_sales_price)
- from catalog_sales
- ,customer
- ,customer_address
- ,date_dim
- where cs_bill_customer_sk = c_customer_sk
-   and c_current_addr_sk = ca_address_sk 
-   and ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475',
-   '85392', '85460', '80348', '81792')
- or ca_state in ('CA','WA','GA')
- or cs_sales_price > 500)
-   and cs_sold_date_sk = d_date_sk
-   and d_qoy = 2 and d_year = 2000
- group by ca_zip
- order by ca_zip
- limit 100
-POSTHOOK: type: QUERY
-Plan optimized by CBO.
-
-Vertex dependency in root stage
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 6 (SIMPLE_EDGE)
-Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
-Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
-Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
-Reducer 8 <- Map 7 (SIMPLE_EDGE), Map 9 (SIMPLE_EDGE)
-
-Stage-0
-  Fetch Operator
-limit:100
-Stage-1
-  Reducer 5
-  File Output Operator [FS_31]
-Limit [LIM_30] (rows=100 width=135)
-  Number of rows:100
-  Select Operator [SEL_29] (rows=174233858 width=135)
-Output:["_col0","_col1"]
-  <-Reducer 4 [SIMPLE_EDGE]
-SHUFFLE [RS_28]
-  Group By Operator [GBY_26] (rows=174233858 width=135)
-
Output:["_col0","_col1"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0
-  <-Reducer 3 [SIMPLE_EDGE]
-SHUFFLE [RS_25]
-  PartitionCols:_col0
-  Group By Operator [GBY_24] (rows=348467716 width=135)
-
Output:["_col0","_col1"],aggregations:["sum(_col7)"],keys:_col4
-Select Operator [SEL_23] (rows=348467716 width=135)
-  Output:["_col4","_col7"]
-  Filter Operator [FIL_22] (rows=348467716 width=135)
-predicate:((_col3) IN ('CA', 'WA', 'GA') or (_col7 > 
500) or (substr(_col4, 1, 5)) IN ('85669', '86197', '88274', '83405', '86475', 
'85392', '85460', '80348', '81792'))
-Merge Join Operator [MERGEJOIN_46] (rows=348467716 
width=135)
-  
Conds:RS_19._col0=RS_20._col1(Inner),Output:["_col3","_col4","_col7"]
-<-Reducer 2 [SIMPLE_EDGE]
-  SHUFFLE [RS_19]
-PartitionCols:_col0
-Merge Join Operator [MERGEJOIN_44] (rows=8801 
width=860)
-  
Conds:RS_16._col1=RS_17._col0(Inner),Output:["_col0","_col3","_col4"]
-<-Map 1 [SIMPLE_EDGE]
-  SHUFFLE [RS_16]
-PartitionCols:_col1
-Select Operator [SEL_2] (rows=8000 
width=860)
-  Output:["_col0","_col1"]
-  Filter Operator [FIL_40] (rows=8000 
width=860)
-predicate:(c_current_addr_sk is not null 
and c_customer_sk is not null)
-TableScan [TS_0] (rows=8000 width=860)
-  
default@customer,customer,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk","c_current_addr_sk"]
-<-Map 6 [SIMPLE_EDGE]
-  SHUFFLE [RS_17]
-PartitionCols:_col0
-Select Operator [SEL_5] (rows=4000 
width=1014)
-  Output:["_col0","_col1","_col2"]
-  Filter Operator [FIL_41] (rows=4000 
width=1014)
-predicate:ca_address_sk is not null
-TableScan [TS_3] (rows=4000 width=1014)
-  
defa

[18/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query15.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query15.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query15.q.out
new file mode 100644
index 000..cf6cf16
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query15.q.out
@@ -0,0 +1,119 @@
+PREHOOK: query: explain
+select  ca_zip
+   ,sum(cs_sales_price)
+ from catalog_sales
+ ,customer
+ ,customer_address
+ ,date_dim
+ where cs_bill_customer_sk = c_customer_sk
+   and c_current_addr_sk = ca_address_sk 
+   and ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475',
+   '85392', '85460', '80348', '81792')
+ or ca_state in ('CA','WA','GA')
+ or cs_sales_price > 500)
+   and cs_sold_date_sk = d_date_sk
+   and d_qoy = 2 and d_year = 2000
+ group by ca_zip
+ order by ca_zip
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select  ca_zip
+   ,sum(cs_sales_price)
+ from catalog_sales
+ ,customer
+ ,customer_address
+ ,date_dim
+ where cs_bill_customer_sk = c_customer_sk
+   and c_current_addr_sk = ca_address_sk 
+   and ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475',
+   '85392', '85460', '80348', '81792')
+ or ca_state in ('CA','WA','GA')
+ or cs_sales_price > 500)
+   and cs_sold_date_sk = d_date_sk
+   and d_qoy = 2 and d_year = 2000
+ group by ca_zip
+ order by ca_zip
+ limit 100
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 6 (SIMPLE_EDGE)
+Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
+Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
+Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
+Reducer 8 <- Map 7 (SIMPLE_EDGE), Map 9 (SIMPLE_EDGE)
+
+Stage-0
+  Fetch Operator
+limit:100
+Stage-1
+  Reducer 5
+  File Output Operator [FS_31]
+Limit [LIM_30] (rows=100 width=135)
+  Number of rows:100
+  Select Operator [SEL_29] (rows=174233858 width=135)
+Output:["_col0","_col1"]
+  <-Reducer 4 [SIMPLE_EDGE]
+SHUFFLE [RS_28]
+  Group By Operator [GBY_26] (rows=174233858 width=135)
+
Output:["_col0","_col1"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0
+  <-Reducer 3 [SIMPLE_EDGE]
+SHUFFLE [RS_25]
+  PartitionCols:_col0
+  Group By Operator [GBY_24] (rows=348467716 width=135)
+
Output:["_col0","_col1"],aggregations:["sum(_col7)"],keys:_col4
+Select Operator [SEL_23] (rows=348467716 width=135)
+  Output:["_col4","_col7"]
+  Filter Operator [FIL_22] (rows=348467716 width=135)
+predicate:((_col3) IN ('CA', 'WA', 'GA') or (_col7 > 
500) or (substr(_col4, 1, 5)) IN ('85669', '86197', '88274', '83405', '86475', 
'85392', '85460', '80348', '81792'))
+Merge Join Operator [MERGEJOIN_46] (rows=348467716 
width=135)
+  
Conds:RS_19._col0=RS_20._col1(Inner),Output:["_col3","_col4","_col7"]
+<-Reducer 2 [SIMPLE_EDGE]
+  SHUFFLE [RS_19]
+PartitionCols:_col0
+Merge Join Operator [MERGEJOIN_44] (rows=8801 
width=860)
+  
Conds:RS_16._col1=RS_17._col0(Inner),Output:["_col0","_col3","_col4"]
+<-Map 1 [SIMPLE_EDGE]
+  SHUFFLE [RS_16]
+PartitionCols:_col1
+Select Operator [SEL_2] (rows=8000 
width=860)
+  Output:["_col0","_col1"]
+  Filter Operator [FIL_40] (rows=8000 
width=860)
+predicate:(c_current_addr_sk is not null 
and c_customer_sk is not null)
+TableScan [TS_0] (rows=8000 width=860)
+  
default@customer,customer,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk","c_current_addr_sk"]
+<-Map 6 [SIMPLE_EDGE]
+  SHUFFLE [RS_17]
+PartitionCols:_col0
+Select Operator [SEL_5] (rows=4000 
width=1014)
+  Output:["_col0","_col1","_col2"]
+  Filter Operator [FIL_41] (rows=4000 
width=1014)
+predicate:ca_address_sk is not null
+TableScan [TS_3] (rows=4000 width=1014)
+

[14/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query39.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query39.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query39.q.out
new file mode 100644
index 000..8e91494
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query39.q.out
@@ -0,0 +1,255 @@
+PREHOOK: query: explain
+with inv as
+(select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy
+   ,stdev,mean, case mean when 0 then null else stdev/mean end cov
+ from(select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy
+,stddev_samp(inv_quantity_on_hand) stdev,avg(inv_quantity_on_hand) 
mean
+  from inventory
+  ,item
+  ,warehouse
+  ,date_dim
+  where inv_item_sk = i_item_sk
+and inv_warehouse_sk = w_warehouse_sk
+and inv_date_sk = d_date_sk
+and d_year =1999
+  group by w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy) foo
+ where case mean when 0 then 0 else stdev/mean end > 1)
+select inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean, inv1.cov
+,inv2.w_warehouse_sk,inv2.i_item_sk,inv2.d_moy,inv2.mean, inv2.cov
+from inv inv1,inv inv2
+where inv1.i_item_sk = inv2.i_item_sk
+  and inv1.w_warehouse_sk =  inv2.w_warehouse_sk
+  and inv1.d_moy=4
+  and inv2.d_moy=4+1
+order by inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean,inv1.cov
+,inv2.d_moy,inv2.mean, inv2.cov
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with inv as
+(select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy
+   ,stdev,mean, case mean when 0 then null else stdev/mean end cov
+ from(select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy
+,stddev_samp(inv_quantity_on_hand) stdev,avg(inv_quantity_on_hand) 
mean
+  from inventory
+  ,item
+  ,warehouse
+  ,date_dim
+  where inv_item_sk = i_item_sk
+and inv_warehouse_sk = w_warehouse_sk
+and inv_date_sk = d_date_sk
+and d_year =1999
+  group by w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy) foo
+ where case mean when 0 then 0 else stdev/mean end > 1)
+select inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean, inv1.cov
+,inv2.w_warehouse_sk,inv2.i_item_sk,inv2.d_moy,inv2.mean, inv2.cov
+from inv inv1,inv inv2
+where inv1.i_item_sk = inv2.i_item_sk
+  and inv1.w_warehouse_sk =  inv2.w_warehouse_sk
+  and inv1.d_moy=4
+  and inv2.d_moy=4+1
+order by inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean,inv1.cov
+,inv2.d_moy,inv2.mean, inv2.cov
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 10 <- Map 14 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
+Reducer 11 <- Reducer 10 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 12 (SIMPLE_EDGE)
+Reducer 3 <- Map 13 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Map 14 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
+Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
+Reducer 6 <- Reducer 11 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
+Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
+Reducer 8 <- Map 1 (SIMPLE_EDGE), Map 12 (SIMPLE_EDGE)
+Reducer 9 <- Map 13 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
+
+Stage-0
+  Fetch Operator
+limit:-1
+Stage-1
+  Reducer 7
+  File Output Operator [FS_63]
+Select Operator [SEL_61] (rows=13756683 width=15)
+  
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9"]
+<-Reducer 6 [SIMPLE_EDGE]
+  SHUFFLE [RS_60]
+Select Operator [SEL_59] (rows=13756683 width=15)
+  
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7"]
+  Merge Join Operator [MERGEJOIN_104] (rows=13756683 width=15)
+Conds:RS_56._col1, _col2=RS_57._col1, 
_col2(Inner),Output:["_col1","_col2","_col3","_col4","_col6","_col7","_col8","_col9"]
+  <-Reducer 11 [SIMPLE_EDGE]
+SHUFFLE [RS_57]
+  PartitionCols:_col1, _col2
+  Select Operator [SEL_55] (rows=12506076 width=15)
+Output:["_col1","_col2","_col3","_col4"]
+Filter Operator [FIL_54] (rows=12506076 width=15)
+  predicate:CASE WHEN ((_col4 = 0.0)) THEN (false) ELSE 
(((_col3 / _col4) > 1.0)) END
+  Select Operator [SEL_53] (rows=25012152 width=15)
+Output:["_col1","_col2","_col3","_col4"]
+Group By Operator [GBY_52] (rows=25012152 width=15)
+  
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["stddev_samp(VALUE._col0)","avg(VALUE._col1)"],keys:KEY._col0,
 KEY._col1, KEY._col2
+<-Reducer 10 [SIMPLE_EDGE]
+  SHUFFLE [RS_51]
+PartitionCols:_col0, _col1, _col2
+Group By

[03/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query87.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query87.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query87.q.out
new file mode 100644
index 000..8430e8a
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query87.q.out
@@ -0,0 +1,239 @@
+PREHOOK: query: explain
+select count(*) 
+from ((select distinct c_last_name, c_first_name, d_date
+   from store_sales, date_dim, customer
+   where store_sales.ss_sold_date_sk = date_dim.d_date_sk
+ and store_sales.ss_customer_sk = customer.c_customer_sk
+ and d_month_seq between 1212 and 1212+11)
+   except
+  (select distinct c_last_name, c_first_name, d_date
+   from catalog_sales, date_dim, customer
+   where catalog_sales.cs_sold_date_sk = date_dim.d_date_sk
+ and catalog_sales.cs_bill_customer_sk = customer.c_customer_sk
+ and d_month_seq between 1212 and 1212+11)
+   except
+  (select distinct c_last_name, c_first_name, d_date
+   from web_sales, date_dim, customer
+   where web_sales.ws_sold_date_sk = date_dim.d_date_sk
+ and web_sales.ws_bill_customer_sk = customer.c_customer_sk
+ and d_month_seq between 1212 and 1212+11)
+) cool_cust
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select count(*) 
+from ((select distinct c_last_name, c_first_name, d_date
+   from store_sales, date_dim, customer
+   where store_sales.ss_sold_date_sk = date_dim.d_date_sk
+ and store_sales.ss_customer_sk = customer.c_customer_sk
+ and d_month_seq between 1212 and 1212+11)
+   except
+  (select distinct c_last_name, c_first_name, d_date
+   from catalog_sales, date_dim, customer
+   where catalog_sales.cs_sold_date_sk = date_dim.d_date_sk
+ and catalog_sales.cs_bill_customer_sk = customer.c_customer_sk
+ and d_month_seq between 1212 and 1212+11)
+   except
+  (select distinct c_last_name, c_first_name, d_date
+   from web_sales, date_dim, customer
+   where web_sales.ws_sold_date_sk = date_dim.d_date_sk
+ and web_sales.ws_bill_customer_sk = customer.c_customer_sk
+ and d_month_seq between 1212 and 1212+11)
+) cool_cust
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 11 <- Map 10 (SIMPLE_EDGE), Map 18 (SIMPLE_EDGE)
+Reducer 12 <- Map 17 (SIMPLE_EDGE), Reducer 11 (SIMPLE_EDGE)
+Reducer 13 <- Reducer 12 (SIMPLE_EDGE), Union 5 (CONTAINS)
+Reducer 14 <- Map 10 (SIMPLE_EDGE), Map 19 (SIMPLE_EDGE)
+Reducer 15 <- Map 17 (SIMPLE_EDGE), Reducer 14 (SIMPLE_EDGE)
+Reducer 16 <- Reducer 15 (SIMPLE_EDGE), Union 7 (CONTAINS)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 10 (SIMPLE_EDGE)
+Reducer 3 <- Map 17 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Reducer 3 (SIMPLE_EDGE), Union 5 (CONTAINS)
+Reducer 6 <- Union 5 (SIMPLE_EDGE), Union 7 (CONTAINS)
+Reducer 8 <- Union 7 (SIMPLE_EDGE)
+Reducer 9 <- Reducer 8 (CUSTOM_SIMPLE_EDGE)
+
+Stage-0
+  Fetch Operator
+limit:-1
+Stage-1
+  Reducer 9
+  File Output Operator [FS_104]
+Group By Operator [GBY_102] (rows=1 width=24)
+  Output:["_col0"],aggregations:["count(VALUE._col0)"]
+<-Reducer 8 [CUSTOM_SIMPLE_EDGE]
+  PARTITION_ONLY_SHUFFLE [RS_101]
+Group By Operator [GBY_100] (rows=1 width=24)
+  Output:["_col0"],aggregations:["count()"]
+  Select Operator [SEL_98] (rows=4537552 width=129)
+Filter Operator [FIL_97] (rows=4537552 width=129)
+  predicate:(((_col3 * 2) = _col4) and (_col3 > 0))
+  Select Operator [SEL_129] (rows=27225312 width=129)
+Output:["_col3","_col4"]
+Group By Operator [GBY_96] (rows=27225312 width=129)
+  
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["sum(VALUE._col0)","sum(VALUE._col1)"],keys:KEY._col0,
 KEY._col1, KEY._col2
+<-Union 7 [SIMPLE_EDGE]
+  <-Reducer 16 [CONTAINS]
+Reduce Output Operator [RS_95]
+  PartitionCols:_col0, _col1, _col2
+  Group By Operator [GBY_94] (rows=54450625 width=129)
+
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["sum(_col3)","sum(_col4)"],keys:_col0,
 _col1, _col2
+Select Operator [SEL_92] (rows=54450625 width=129)
+  Output:["_col0","_col1","_col2","_col3","_col4"]
+  Select Operator [SEL_89] (rows=43560808 
width=135)
+
Output:["_col0","_col1","_col2","_col3","_col4"]
+Group By Operator [GBY_88] (rows=43560808 
width=135)
+

[39/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query14.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query14.q.out 
b/ql/src/test/results/clientpositive/perf/query14.q.out
deleted file mode 100644
index dac6e0f..000
--- a/ql/src/test/results/clientpositive/perf/query14.q.out
+++ /dev/null
@@ -1,1286 +0,0 @@
-Warning: Shuffle Join MERGEJOIN[892][tables = [$hdt$_2, $hdt$_3, $hdt$_1]] in 
Stage 'Reducer 16' is a cross product
-Warning: Shuffle Join MERGEJOIN[890][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in 
Stage 'Reducer 5' is a cross product
-Warning: Shuffle Join MERGEJOIN[891][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in 
Stage 'Reducer 12' is a cross product
-PREHOOK: query: explain
-with  cross_items as
- (select i_item_sk ss_item_sk
- from item,
- (select iss.i_brand_id brand_id
- ,iss.i_class_id class_id
- ,iss.i_category_id category_id
- from store_sales
- ,item iss
- ,date_dim d1
- where ss_item_sk = iss.i_item_sk
-   and ss_sold_date_sk = d1.d_date_sk
-   and d1.d_year between 1999 AND 1999 + 2
- intersect 
- select ics.i_brand_id
- ,ics.i_class_id
- ,ics.i_category_id
- from catalog_sales
- ,item ics
- ,date_dim d2
- where cs_item_sk = ics.i_item_sk
-   and cs_sold_date_sk = d2.d_date_sk
-   and d2.d_year between 1999 AND 1999 + 2
- intersect
- select iws.i_brand_id
- ,iws.i_class_id
- ,iws.i_category_id
- from web_sales
- ,item iws
- ,date_dim d3
- where ws_item_sk = iws.i_item_sk
-   and ws_sold_date_sk = d3.d_date_sk
-   and d3.d_year between 1999 AND 1999 + 2) x
- where i_brand_id = brand_id
-  and i_class_id = class_id
-  and i_category_id = category_id
-),
- avg_sales as
- (select avg(quantity*list_price) average_sales
-  from (select ss_quantity quantity
- ,ss_list_price list_price
-   from store_sales
-   ,date_dim
-   where ss_sold_date_sk = d_date_sk
- and d_year between 1999 and 2001 
-   union all 
-   select cs_quantity quantity 
- ,cs_list_price list_price
-   from catalog_sales
-   ,date_dim
-   where cs_sold_date_sk = d_date_sk
- and d_year between 1998 and 1998 + 2 
-   union all
-   select ws_quantity quantity
- ,ws_list_price list_price
-   from web_sales
-   ,date_dim
-   where ws_sold_date_sk = d_date_sk
- and d_year between 1998 and 1998 + 2) x)
-  select  channel, i_brand_id,i_class_id,i_category_id,sum(sales), 
sum(number_sales)
- from(
-   select 'store' channel, i_brand_id,i_class_id
- ,i_category_id,sum(ss_quantity*ss_list_price) sales
- , count(*) number_sales
-   from store_sales
-   ,item
-   ,date_dim
-   where ss_item_sk in (select ss_item_sk from cross_items)
- and ss_item_sk = i_item_sk
- and ss_sold_date_sk = d_date_sk
- and d_year = 1998+2 
- and d_moy = 11
-   group by i_brand_id,i_class_id,i_category_id
-   having sum(ss_quantity*ss_list_price) > (select average_sales from 
avg_sales)
-   union all
-   select 'catalog' channel, i_brand_id,i_class_id,i_category_id, 
sum(cs_quantity*cs_list_price) sales, count(*) number_sales
-   from catalog_sales
-   ,item
-   ,date_dim
-   where cs_item_sk in (select ss_item_sk from cross_items)
- and cs_item_sk = i_item_sk
- and cs_sold_date_sk = d_date_sk
- and d_year = 1998+2 
- and d_moy = 11
-   group by i_brand_id,i_class_id,i_category_id
-   having sum(cs_quantity*cs_list_price) > (select average_sales from 
avg_sales)
-   union all
-   select 'web' channel, i_brand_id,i_class_id,i_category_id, 
sum(ws_quantity*ws_list_price) sales , count(*) number_sales
-   from web_sales
-   ,item
-   ,date_dim
-   where ws_item_sk in (select ss_item_sk from cross_items)
- and ws_item_sk = i_item_sk
- and ws_sold_date_sk = d_date_sk
- and d_year = 1998+2
- and d_moy = 11
-   group by i_brand_id,i_class_id,i_category_id
-   having sum(ws_quantity*ws_list_price) > (select average_sales from 
avg_sales)
- ) y
- group by rollup (channel, i_brand_id,i_class_id,i_category_id)
- order by channel,i_brand_id,i_class_id,i_category_id
- limit 100
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-with  cross_items as
- (select i_item_sk ss_item_sk
- from item,
- (select iss.i_brand_id brand_id
- ,iss.i_class_id class_id
- ,iss.i_category_id category_id
- from store_sales
- ,item iss
- ,date_dim d1
- where ss_item_sk = iss.i_item_sk
-   and ss_sold_date_sk = d1.d_date_sk
-   and d1.d_year between 1999 AND 1999 + 2
- intersect 
- select ics.i_brand_id
- ,ics.i_class_id
- ,ics.i_category_id
- from catalog_sales
- ,item ics
- ,date_dim d2
- where cs_item_sk = ics.i_item_sk
-   and

[05/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query78.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query78.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query78.q.out
new file mode 100644
index 000..91c244f
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query78.q.out
@@ -0,0 +1,292 @@
+PREHOOK: query: explain
+with ws as
+  (select d_year AS ws_sold_year, ws_item_sk,
+ws_bill_customer_sk ws_customer_sk,
+sum(ws_quantity) ws_qty,
+sum(ws_wholesale_cost) ws_wc,
+sum(ws_sales_price) ws_sp
+   from web_sales
+   left join web_returns on wr_order_number=ws_order_number and 
ws_item_sk=wr_item_sk
+   join date_dim on ws_sold_date_sk = d_date_sk
+   where wr_order_number is null
+   group by d_year, ws_item_sk, ws_bill_customer_sk
+   ),
+cs as
+  (select d_year AS cs_sold_year, cs_item_sk,
+cs_bill_customer_sk cs_customer_sk,
+sum(cs_quantity) cs_qty,
+sum(cs_wholesale_cost) cs_wc,
+sum(cs_sales_price) cs_sp
+   from catalog_sales
+   left join catalog_returns on cr_order_number=cs_order_number and 
cs_item_sk=cr_item_sk
+   join date_dim on cs_sold_date_sk = d_date_sk
+   where cr_order_number is null
+   group by d_year, cs_item_sk, cs_bill_customer_sk
+   ),
+ss as
+  (select d_year AS ss_sold_year, ss_item_sk,
+ss_customer_sk,
+sum(ss_quantity) ss_qty,
+sum(ss_wholesale_cost) ss_wc,
+sum(ss_sales_price) ss_sp
+   from store_sales
+   left join store_returns on sr_ticket_number=ss_ticket_number and 
ss_item_sk=sr_item_sk
+   join date_dim on ss_sold_date_sk = d_date_sk
+   where sr_ticket_number is null
+   group by d_year, ss_item_sk, ss_customer_sk
+   )
+ select 
+ss_sold_year, ss_item_sk, ss_customer_sk,
+round(ss_qty/(coalesce(ws_qty+cs_qty,1)),2) ratio,
+ss_qty store_qty, ss_wc store_wholesale_cost, ss_sp store_sales_price,
+coalesce(ws_qty,0)+coalesce(cs_qty,0) other_chan_qty,
+coalesce(ws_wc,0)+coalesce(cs_wc,0) other_chan_wholesale_cost,
+coalesce(ws_sp,0)+coalesce(cs_sp,0) other_chan_sales_price
+from ss
+left join ws on (ws_sold_year=ss_sold_year and ws_item_sk=ss_item_sk and 
ws_customer_sk=ss_customer_sk)
+left join cs on (cs_sold_year=ss_sold_year and cs_item_sk=cs_item_sk and 
cs_customer_sk=ss_customer_sk)
+where coalesce(ws_qty,0)>0 and coalesce(cs_qty, 0)>0 and ss_sold_year=2000
+order by 
+  ss_sold_year, ss_item_sk, ss_customer_sk,
+  ss_qty desc, ss_wc desc, ss_sp desc,
+  other_chan_qty,
+  other_chan_wholesale_cost,
+  other_chan_sales_price,
+  round(ss_qty/(coalesce(ws_qty+cs_qty,1)),2)
+limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with ws as
+  (select d_year AS ws_sold_year, ws_item_sk,
+ws_bill_customer_sk ws_customer_sk,
+sum(ws_quantity) ws_qty,
+sum(ws_wholesale_cost) ws_wc,
+sum(ws_sales_price) ws_sp
+   from web_sales
+   left join web_returns on wr_order_number=ws_order_number and 
ws_item_sk=wr_item_sk
+   join date_dim on ws_sold_date_sk = d_date_sk
+   where wr_order_number is null
+   group by d_year, ws_item_sk, ws_bill_customer_sk
+   ),
+cs as
+  (select d_year AS cs_sold_year, cs_item_sk,
+cs_bill_customer_sk cs_customer_sk,
+sum(cs_quantity) cs_qty,
+sum(cs_wholesale_cost) cs_wc,
+sum(cs_sales_price) cs_sp
+   from catalog_sales
+   left join catalog_returns on cr_order_number=cs_order_number and 
cs_item_sk=cr_item_sk
+   join date_dim on cs_sold_date_sk = d_date_sk
+   where cr_order_number is null
+   group by d_year, cs_item_sk, cs_bill_customer_sk
+   ),
+ss as
+  (select d_year AS ss_sold_year, ss_item_sk,
+ss_customer_sk,
+sum(ss_quantity) ss_qty,
+sum(ss_wholesale_cost) ss_wc,
+sum(ss_sales_price) ss_sp
+   from store_sales
+   left join store_returns on sr_ticket_number=ss_ticket_number and 
ss_item_sk=sr_item_sk
+   join date_dim on ss_sold_date_sk = d_date_sk
+   where sr_ticket_number is null
+   group by d_year, ss_item_sk, ss_customer_sk
+   )
+ select 
+ss_sold_year, ss_item_sk, ss_customer_sk,
+round(ss_qty/(coalesce(ws_qty+cs_qty,1)),2) ratio,
+ss_qty store_qty, ss_wc store_wholesale_cost, ss_sp store_sales_price,
+coalesce(ws_qty,0)+coalesce(cs_qty,0) other_chan_qty,
+coalesce(ws_wc,0)+coalesce(cs_wc,0) other_chan_wholesale_cost,
+coalesce(ws_sp,0)+coalesce(cs_sp,0) other_chan_sales_price
+from ss
+left join ws on (ws_sold_year=ss_sold_year and ws_item_sk=ss_item_sk and 
ws_customer_sk=ss_customer_sk)
+left join cs on (cs_sold_year=ss_sold_year and cs_item_sk=cs_item_sk and 
cs_customer_sk=ss_customer_sk)
+where coalesce(ws_qty,0)>0 and coalesce(cs_qty, 0)>0 and ss_sold_year=2000
+order by 
+  ss_sold_year, ss_item_sk, ss_customer_sk,
+  ss_qty desc, ss_wc desc, ss_sp desc,
+  other_chan_qty,
+  other_chan_wholesale_cost,
+  other_chan_sales_price,
+  round(ss_qty/(coalesce(ws_qty+cs_qty,1)),2)
+limit 100
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency

[13/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/tez/query45.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/tez/query45.q.out 
b/ql/src/test/results/clientpositive/perf/tez/query45.q.out
new file mode 100644
index 000..3efed2e
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/tez/query45.q.out
@@ -0,0 +1,180 @@
+Warning: Shuffle Join MERGEJOIN[87][tables = [$hdt$_0, $hdt$_1, $hdt$_2, 
$hdt$_3]] in Stage 'Reducer 4' is a cross product
+PREHOOK: query: explain
+select  ca_zip, ca_county, sum(ws_sales_price)
+ from web_sales, customer, customer_address, date_dim, item
+ where ws_bill_customer_sk = c_customer_sk
+   and c_current_addr_sk = ca_address_sk 
+   and ws_item_sk = i_item_sk 
+   and ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475', 
'85392', '85460', '80348', '81792')
+ or 
+ i_item_id in (select i_item_id
+ from item
+ where i_item_sk in (2, 3, 5, 7, 11, 13, 17, 19, 
23, 29)
+ )
+   )
+   and ws_sold_date_sk = d_date_sk
+   and d_qoy = 2 and d_year = 2000
+ group by ca_zip, ca_county
+ order by ca_zip, ca_county
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select  ca_zip, ca_county, sum(ws_sales_price)
+ from web_sales, customer, customer_address, date_dim, item
+ where ws_bill_customer_sk = c_customer_sk
+   and c_current_addr_sk = ca_address_sk 
+   and ws_item_sk = i_item_sk 
+   and ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475', 
'85392', '85460', '80348', '81792')
+ or 
+ i_item_id in (select i_item_id
+ from item
+ where i_item_sk in (2, 3, 5, 7, 11, 13, 17, 19, 
23, 29)
+ )
+   )
+   and ws_sold_date_sk = d_date_sk
+   and d_qoy = 2 and d_year = 2000
+ group by ca_zip, ca_county
+ order by ca_zip, ca_county
+ limit 100
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 10 <- Reducer 14 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
+Reducer 11 <- Map 8 (SIMPLE_EDGE)
+Reducer 12 <- Map 8 (CUSTOM_SIMPLE_EDGE)
+Reducer 14 <- Map 13 (SIMPLE_EDGE), Map 15 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 7 (SIMPLE_EDGE)
+Reducer 3 <- Reducer 10 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Reducer 12 (CUSTOM_SIMPLE_EDGE), Reducer 3 (CUSTOM_SIMPLE_EDGE)
+Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
+Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
+Reducer 9 <- Map 8 (SIMPLE_EDGE), Reducer 11 (ONE_TO_ONE_EDGE)
+
+Stage-0
+  Fetch Operator
+limit:100
+Stage-1
+  Reducer 6
+  File Output Operator [FS_59]
+Limit [LIM_58] (rows=100 width=152)
+  Number of rows:100
+  Select Operator [SEL_57] (rows=95833781 width=152)
+Output:["_col0","_col1","_col2"]
+  <-Reducer 5 [SIMPLE_EDGE]
+SHUFFLE [RS_56]
+  Group By Operator [GBY_54] (rows=95833781 width=152)
+
Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0,
 KEY._col1
+  <-Reducer 4 [SIMPLE_EDGE]
+SHUFFLE [RS_53]
+  PartitionCols:_col0, _col1
+  Group By Operator [GBY_52] (rows=191667562 width=152)
+
Output:["_col0","_col1","_col2"],aggregations:["sum(_col3)"],keys:_col8, _col7
+Select Operator [SEL_51] (rows=191667562 width=152)
+  Output:["_col3","_col7","_col8"]
+  Filter Operator [FIL_50] (rows=191667562 width=152)
+predicate:((substr(_col8, 1, 5)) IN ('85669', '86197', 
'88274', '83405', '86475', '85392', '85460', '80348', '81792') or CASE WHEN 
((_col14 = 0)) THEN (false) WHEN (_col17 is not null) THEN (true) WHEN (_col13 
is null) THEN (null) WHEN ((_col15 < _col14)) THEN (null) ELSE (false) END)
+Select Operator [SEL_49] (rows=191667562 width=152)
+  
Output:["_col3","_col7","_col8","_col13","_col14","_col15","_col17"]
+  Merge Join Operator [MERGEJOIN_87] (rows=191667562 
width=152)
+
Conds:(Inner),Output:["_col3","_col4","_col6","_col8","_col12","_col16","_col17"]
+  <-Reducer 12 [CUSTOM_SIMPLE_EDGE]
+PARTITION_ONLY_SHUFFLE [RS_47]
+  Group By Operator [GBY_38] (rows=1 width=16)
+
Output:["_col0","_col1"],aggregations:["count(VALUE._col0)","count(VALUE._col1)"]
+  <-Map 8 [CUSTOM_SIMPLE_EDGE]
+SHUFFLE [RS_37]
+  Group By Operator [GBY_36] (rows=1 width=16)
+

[37/41] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary) (addendum)

2017-09-29 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/9244fdc7/ql/src/test/results/clientpositive/perf/query22.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/query22.q.out 
b/ql/src/test/results/clientpositive/perf/query22.q.out
deleted file mode 100644
index 995f61d..000
--- a/ql/src/test/results/clientpositive/perf/query22.q.out
+++ /dev/null
@@ -1,121 +0,0 @@
-PREHOOK: query: explain
-select  i_product_name
- ,i_brand
- ,i_class
- ,i_category
- ,avg(inv_quantity_on_hand) qoh
-   from inventory
-   ,date_dim
-   ,item
-   ,warehouse
-   where inv_date_sk=d_date_sk
-  and inv_item_sk=i_item_sk
-  and inv_warehouse_sk = w_warehouse_sk
-  and d_month_seq between 1212 and 1212 + 11
-   group by rollup(i_product_name
-   ,i_brand
-   ,i_class
-   ,i_category)
-order by qoh, i_product_name, i_brand, i_class, i_category
-limit 100
-PREHOOK: type: QUERY
-POSTHOOK: query: explain
-select  i_product_name
- ,i_brand
- ,i_class
- ,i_category
- ,avg(inv_quantity_on_hand) qoh
-   from inventory
-   ,date_dim
-   ,item
-   ,warehouse
-   where inv_date_sk=d_date_sk
-  and inv_item_sk=i_item_sk
-  and inv_warehouse_sk = w_warehouse_sk
-  and d_month_seq between 1212 and 1212 + 11
-   group by rollup(i_product_name
-   ,i_brand
-   ,i_class
-   ,i_category)
-order by qoh, i_product_name, i_brand, i_class, i_category
-limit 100
-POSTHOOK: type: QUERY
-Plan optimized by CBO.
-
-Vertex dependency in root stage
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 7 (SIMPLE_EDGE)
-Reducer 3 <- Map 8 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Map 9 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
-Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
-Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
-
-Stage-0
-  Fetch Operator
-limit:100
-Stage-1
-  Reducer 6
-  File Output Operator [FS_30]
-Limit [LIM_29] (rows=100 width=15)
-  Number of rows:100
-  Select Operator [SEL_28] (rows=125060762 width=15)
-Output:["_col0","_col1","_col2","_col3","_col4"]
-  <-Reducer 5 [SIMPLE_EDGE]
-SHUFFLE [RS_27]
-  Select Operator [SEL_25] (rows=125060762 width=15)
-Output:["_col0","_col1","_col2","_col3","_col4"]
-Group By Operator [GBY_24] (rows=125060762 width=15)
-  
Output:["_col0","_col1","_col2","_col3","_col5"],aggregations:["avg(VALUE._col0)"],keys:KEY._col0,
 KEY._col1, KEY._col2, KEY._col3, KEY._col4
-<-Reducer 4 [SIMPLE_EDGE]
-  SHUFFLE [RS_23]
-PartitionCols:_col0, _col1, _col2, _col3, _col4
-Group By Operator [GBY_22] (rows=250121525 width=15)
-  
Output:["_col0","_col1","_col2","_col3","_col4","_col5"],aggregations:["avg(_col3)"],keys:_col8,
 _col9, _col10, _col11, 0
-  Merge Join Operator [MERGEJOIN_46] (rows=50024305 
width=15)
-
Conds:RS_18._col1=RS_19._col0(Inner),Output:["_col3","_col8","_col9","_col10","_col11"]
-  <-Map 9 [SIMPLE_EDGE]
-SHUFFLE [RS_19]
-  PartitionCols:_col0
-  Select Operator [SEL_11] (rows=462000 width=1436)
-Output:["_col0","_col1","_col2","_col3","_col4"]
-Filter Operator [FIL_43] (rows=462000 width=1436)
-  predicate:i_item_sk is not null
-  TableScan [TS_9] (rows=462000 width=1436)
-
default@item,item,Tbl:COMPLETE,Col:NONE,Output:["i_item_sk","i_brand","i_class","i_category","i_product_name"]
-  <-Reducer 3 [SIMPLE_EDGE]
-SHUFFLE [RS_18]
-  PartitionCols:_col1
-  Merge Join Operator [MERGEJOIN_45] (rows=45476640 
width=15)
-
Conds:RS_15._col2=RS_16._col0(Inner),Output:["_col1","_col3"]
-  <-Map 8 [SIMPLE_EDGE]
-SHUFFLE [RS_16]
-  PartitionCols:_col0
-  Select Operator [SEL_8] (rows=27 width=1029)
-Output:["_col0"]
-Filter Operator [FIL_42] (rows=27 width=1029)
-  predicate:w_warehouse_sk is not null
-  TableScan [TS_6] (rows=27 width=1029)
-
default@warehouse,warehouse,Tbl:COMPLETE,Col:NONE,Output:["w_warehouse_sk"]
-

hive git commit: HIVE-17563: CodahaleMetrics.JsonFileReporter is not updating hive.service.metrics.file.location (Alexander Kolbasov, reviewed by Sahil Takiar)

2017-09-29 Thread stakiar

Repository: hive
Updated Branches:
  refs/heads/master 23478cfeb -> 7dc701c59


HIVE-17563: CodahaleMetrics.JsonFileReporter is not updating 
hive.service.metrics.file.location (Alexander Kolbasov, reviewed by Sahil 
Takiar)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/7dc701c5
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/7dc701c5
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/7dc701c5

Branch: refs/heads/master
Commit: 7dc701c592d2083c2e05f06172788c18912d71ae
Parents: 23478cf
Author: Alexander Kolbasov 
Authored: Fri Sep 29 16:51:01 2017 -0700
Committer: Sahil Takiar 
Committed: Fri Sep 29 16:51:32 2017 -0700

--
 .../metrics2/JsonFileMetricsReporter.java   | 192 +++
 .../metrics/metrics2/TestCodahaleMetrics.java   |  86 ++---
 .../hive/metastore/metrics/JsonReporter.java| 131 -
 .../hive/metastore/metrics/TestMetrics.java |  75 ++--
 4 files changed, 277 insertions(+), 207 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/hive/blob/7dc701c5/common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
--
diff --git 
a/common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
 
b/common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
index c07517a..96243cb 100644
--- 
a/common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
+++ 
b/common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
@@ -23,114 +23,156 @@ import com.codahale.metrics.json.MetricsModule;
 import com.fasterxml.jackson.core.JsonProcessingException;
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.fasterxml.jackson.databind.ObjectWriter;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
 import java.io.BufferedWriter;
+import java.io.FileWriter;
 import java.io.IOException;
-import java.io.OutputStreamWriter;
-import java.net.URI;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.nio.file.attribute.FileAttribute;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.Set;
 import java.util.concurrent.Executors;
 import java.util.concurrent.ScheduledExecutorService;
 import java.util.concurrent.TimeUnit;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.fs.permission.FsPermission;
-import org.apache.hadoop.hive.conf.HiveConf;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
 
 
 /**
- * A metrics reporter for CodahaleMetrics that dumps metrics periodically into 
a file in JSON format.
+ * A metrics reporter for CodahaleMetrics that dumps metrics periodically into
+ * a file in JSON format. Only files on local filesystems are supported.
  */
-
-public class JsonFileMetricsReporter implements CodahaleReporter {
+public class JsonFileMetricsReporter implements CodahaleReporter, Runnable {
+  //
+  // Implementation notes.
+  //
+  // 1. Since only local file systems are supported, there is no need to use 
Hadoop
+  //version of Path class.
+  // 2. java.nio package provides modern implementation of file and directory 
operations
+  //which is better then the traditional java.io, so we are using it here.
+  //In particular, it supports atomic creation of temporary files with 
specified
+  //permissions in the specified directory. This also avoids various 
attacks possible
+  //when temp file name is generated first, followed by file creation.
+  //See http://www.oracle.com/technetwork/articles/javase/nio-139333.html 
for
+  //the description of NIO API and
+  //http://docs.oracle.com/javase/tutorial/essential/io/legacy.html for the
+  //description of interoperability between legacy IO api vs NIO API.
+  // 3. To avoid race conditions with readers of the metrics file, the 
implementation
+  //dumps metrics to a temporary file in the same directory as the actual 
metrics
+  //file and then renames it to the destination. Since both are located on 
the same
+  //filesystem, this rename is likely to be atomic (as long as the 
underlying OS
+  //support atomic renames.
+  //
+  // NOTE: This reporter is very similar to
+  //   org.apache.hadoop.hive.metastore.metrics.JsonReporter.
+  //   It would be good to unify the two.
+  //
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(JsonFileMetricsReporter.class);

[27/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query31.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query31.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query31.q.out
new file mode 100644
index 000..6ddec52
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query31.q.out
@@ -0,0 +1,815 @@
+PREHOOK: query: explain
+with ss as
+ (select ca_county,d_qoy, d_year,sum(ss_ext_sales_price) as store_sales
+ from store_sales,date_dim,customer_address
+ where ss_sold_date_sk = d_date_sk
+  and ss_addr_sk=ca_address_sk
+ group by ca_county,d_qoy, d_year),
+ ws as
+ (select ca_county,d_qoy, d_year,sum(ws_ext_sales_price) as web_sales
+ from web_sales,date_dim,customer_address
+ where ws_sold_date_sk = d_date_sk
+  and ws_bill_addr_sk=ca_address_sk
+ group by ca_county,d_qoy, d_year)
+ select /* tt */
+ss1.ca_county
+   ,ss1.d_year
+   ,ws2.web_sales/ws1.web_sales web_q1_q2_increase
+   ,ss2.store_sales/ss1.store_sales store_q1_q2_increase
+   ,ws3.web_sales/ws2.web_sales web_q2_q3_increase
+   ,ss3.store_sales/ss2.store_sales store_q2_q3_increase
+ from
+ss ss1
+   ,ss ss2
+   ,ss ss3
+   ,ws ws1
+   ,ws ws2
+   ,ws ws3
+ where
+ss1.d_qoy = 1
+and ss1.d_year = 2000
+and ss1.ca_county = ss2.ca_county
+and ss2.d_qoy = 2
+and ss2.d_year = 2000
+ and ss2.ca_county = ss3.ca_county
+and ss3.d_qoy = 3
+and ss3.d_year = 2000
+and ss1.ca_county = ws1.ca_county
+and ws1.d_qoy = 1
+and ws1.d_year = 2000
+and ws1.ca_county = ws2.ca_county
+and ws2.d_qoy = 2
+and ws2.d_year = 2000
+and ws1.ca_county = ws3.ca_county
+and ws3.d_qoy = 3
+and ws3.d_year =2000
+and case when ws1.web_sales > 0 then ws2.web_sales/ws1.web_sales else null 
end 
+   > case when ss1.store_sales > 0 then ss2.store_sales/ss1.store_sales 
else null end
+and case when ws2.web_sales > 0 then ws3.web_sales/ws2.web_sales else null 
end
+   > case when ss2.store_sales > 0 then ss3.store_sales/ss2.store_sales 
else null end
+ order by ss1.d_year
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with ss as
+ (select ca_county,d_qoy, d_year,sum(ss_ext_sales_price) as store_sales
+ from store_sales,date_dim,customer_address
+ where ss_sold_date_sk = d_date_sk
+  and ss_addr_sk=ca_address_sk
+ group by ca_county,d_qoy, d_year),
+ ws as
+ (select ca_county,d_qoy, d_year,sum(ws_ext_sales_price) as web_sales
+ from web_sales,date_dim,customer_address
+ where ws_sold_date_sk = d_date_sk
+  and ws_bill_addr_sk=ca_address_sk
+ group by ca_county,d_qoy, d_year)
+ select /* tt */
+ss1.ca_county
+   ,ss1.d_year
+   ,ws2.web_sales/ws1.web_sales web_q1_q2_increase
+   ,ss2.store_sales/ss1.store_sales store_q1_q2_increase
+   ,ws3.web_sales/ws2.web_sales web_q2_q3_increase
+   ,ss3.store_sales/ss2.store_sales store_q2_q3_increase
+ from
+ss ss1
+   ,ss ss2
+   ,ss ss3
+   ,ws ws1
+   ,ws ws2
+   ,ws ws3
+ where
+ss1.d_qoy = 1
+and ss1.d_year = 2000
+and ss1.ca_county = ss2.ca_county
+and ss2.d_qoy = 2
+and ss2.d_year = 2000
+ and ss2.ca_county = ss3.ca_county
+and ss3.d_qoy = 3
+and ss3.d_year = 2000
+and ss1.ca_county = ws1.ca_county
+and ws1.d_qoy = 1
+and ws1.d_year = 2000
+and ws1.ca_county = ws2.ca_county
+and ws2.d_qoy = 2
+and ws2.d_year = 2000
+and ws1.ca_county = ws3.ca_county
+and ws3.d_qoy = 3
+and ws3.d_year =2000
+and case when ws1.web_sales > 0 then ws2.web_sales/ws1.web_sales else null 
end 
+   > case when ss1.store_sales > 0 then ss2.store_sales/ss1.store_sales 
else null end
+and case when ws2.web_sales > 0 then ws3.web_sales/ws2.web_sales else null 
end
+   > case when ss2.store_sales > 0 then ss3.store_sales/ss2.store_sales 
else null end
+ order by ss1.d_year
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+  Stage-1 is a root stage
+  Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+  Stage: Stage-1
+Spark
+  Edges:
+Reducer 10 <- Map 13 (PARTITION-LEVEL SORT, 754), Reducer 9 
(PARTITION-LEVEL SORT, 754)
+Reducer 11 <- Reducer 10 (GROUP, 481)
+Reducer 15 <- Map 14 (PARTITION-LEVEL SORT, 398), Map 18 
(PARTITION-LEVEL SORT, 398)
+Reducer 16 <- Map 19 (PARTITION-LEVEL SORT, 754), Reducer 15 
(PARTITION-LEVEL SORT, 754)
+Reducer 17 <- Reducer 16 (GROUP, 481)
+Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 398), Map 6 (PARTITION-LEVEL 
SORT, 398)
+Reducer 21 <- Map 20 (PARTITION-LEVEL SORT, 154), Map 25 
(PARTITION-LEVEL SORT, 154)
+Reducer 22 <- Map 26 (PARTITION-LEVEL SORT, 486), Reducer 21 
(PARTITION-LEVEL SORT, 486)
+Reducer 23 <- Reducer 22 (GROUP, 186)
+Reducer 24 <- Reducer 23 (PARTITION-LEVEL SORT, 278), Reducer 30 
(PARTITION-LEVEL SORT

[32/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query15.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query15.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query15.q.out
new file mode 100644
index 000..c54b95a
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query15.q.out
@@ -0,0 +1,223 @@
+PREHOOK: query: explain
+select  ca_zip
+   ,sum(cs_sales_price)
+ from catalog_sales
+ ,customer
+ ,customer_address
+ ,date_dim
+ where cs_bill_customer_sk = c_customer_sk
+   and c_current_addr_sk = ca_address_sk 
+   and ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475',
+   '85392', '85460', '80348', '81792')
+ or ca_state in ('CA','WA','GA')
+ or cs_sales_price > 500)
+   and cs_sold_date_sk = d_date_sk
+   and d_qoy = 2 and d_year = 2000
+ group by ca_zip
+ order by ca_zip
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select  ca_zip
+   ,sum(cs_sales_price)
+ from catalog_sales
+ ,customer
+ ,customer_address
+ ,date_dim
+ where cs_bill_customer_sk = c_customer_sk
+   and c_current_addr_sk = ca_address_sk 
+   and ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475',
+   '85392', '85460', '80348', '81792')
+ or ca_state in ('CA','WA','GA')
+ or cs_sales_price > 500)
+   and cs_sold_date_sk = d_date_sk
+   and d_qoy = 2 and d_year = 2000
+ group by ca_zip
+ order by ca_zip
+ limit 100
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+  Stage-1 is a root stage
+  Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+  Stage: Stage-1
+Spark
+  Edges:
+Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 855), Map 6 (PARTITION-LEVEL 
SORT, 855)
+Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 927), Reducer 8 
(PARTITION-LEVEL SORT, 927)
+Reducer 4 <- Reducer 3 (GROUP, 369)
+Reducer 5 <- Reducer 4 (SORT, 1)
+Reducer 8 <- Map 7 (PARTITION-LEVEL SORT, 305), Map 9 (PARTITION-LEVEL 
SORT, 305)
+ A masked pattern was here 
+  Vertices:
+Map 1 
+Map Operator Tree:
+TableScan
+  alias: customer
+  Statistics: Num rows: 8000 Data size: 68801615852 Basic 
stats: COMPLETE Column stats: NONE
+  Filter Operator
+predicate: (c_current_addr_sk is not null and 
c_customer_sk is not null) (type: boolean)
+Statistics: Num rows: 8000 Data size: 68801615852 
Basic stats: COMPLETE Column stats: NONE
+Select Operator
+  expressions: c_customer_sk (type: int), 
c_current_addr_sk (type: int)
+  outputColumnNames: _col0, _col1
+  Statistics: Num rows: 8000 Data size: 68801615852 
Basic stats: COMPLETE Column stats: NONE
+  Reduce Output Operator
+key expressions: _col1 (type: int)
+sort order: +
+Map-reduce partition columns: _col1 (type: int)
+Statistics: Num rows: 8000 Data size: 68801615852 
Basic stats: COMPLETE Column stats: NONE
+value expressions: _col0 (type: int)
+Map 6 
+Map Operator Tree:
+TableScan
+  alias: customer_address
+  Statistics: Num rows: 4000 Data size: 40595195284 Basic 
stats: COMPLETE Column stats: NONE
+  Filter Operator
+predicate: ca_address_sk is not null (type: boolean)
+Statistics: Num rows: 4000 Data size: 40595195284 
Basic stats: COMPLETE Column stats: NONE
+Select Operator
+  expressions: ca_address_sk (type: int), ca_state (type: 
string), ca_zip (type: string)
+  outputColumnNames: _col0, _col1, _col2
+  Statistics: Num rows: 4000 Data size: 40595195284 
Basic stats: COMPLETE Column stats: NONE
+  Reduce Output Operator
+key expressions: _col0 (type: int)
+sort order: +
+Map-reduce partition columns: _col0 (type: int)
+Statistics: Num rows: 4000 Data size: 40595195284 
Basic stats: COMPLETE Column stats: NONE
+value expressions: _col1 (type: string), _col2 (type: 
string)
+Map 7 
+Map Operator Tree:
+TableScan
+  alias: catalog_sales
+  Statistics: Num rows: 287989836 Data size: 38999608952 Basic 
stats: COMPLETE Column stats: NONE
+  Filter Operator
+predicate: (cs_b

[34/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query11.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query11.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query11.q.out
new file mode 100644
index 000..17b2309
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query11.q.out
@@ -0,0 +1,683 @@
+PREHOOK: query: explain
+with year_total as (
+ select c_customer_id customer_id
+   ,c_first_name customer_first_name
+   ,c_last_name customer_last_name
+   ,c_preferred_cust_flag
+   ,c_birth_country customer_birth_country
+   ,c_login customer_login
+   ,c_email_address customer_email_address
+   ,d_year dyear
+   ,sum(ss_ext_list_price-ss_ext_discount_amt) year_total
+   ,'s' sale_type
+ from customer
+ ,store_sales
+ ,date_dim
+ where c_customer_sk = ss_customer_sk
+   and ss_sold_date_sk = d_date_sk
+ group by c_customer_id
+ ,c_first_name
+ ,c_last_name
+ ,d_year
+ ,c_preferred_cust_flag
+ ,c_birth_country
+ ,c_login
+ ,c_email_address
+ ,d_year 
+ union all
+ select c_customer_id customer_id
+   ,c_first_name customer_first_name
+   ,c_last_name customer_last_name
+   ,c_preferred_cust_flag
+   ,c_birth_country customer_birth_country
+   ,c_login customer_login
+   ,c_email_address customer_email_address
+   ,d_year dyear
+   ,sum(ws_ext_list_price-ws_ext_discount_amt) year_total
+   ,'w' sale_type
+ from customer
+ ,web_sales
+ ,date_dim
+ where c_customer_sk = ws_bill_customer_sk
+   and ws_sold_date_sk = d_date_sk
+ group by c_customer_id
+ ,c_first_name
+ ,c_last_name
+ ,c_preferred_cust_flag
+ ,c_birth_country
+ ,c_login
+ ,c_email_address
+ ,d_year
+ )
+  select  t_s_secyear.c_preferred_cust_flag
+ from year_total t_s_firstyear
+ ,year_total t_s_secyear
+ ,year_total t_w_firstyear
+ ,year_total t_w_secyear
+ where t_s_secyear.customer_id = t_s_firstyear.customer_id
+ and t_s_firstyear.customer_id = t_w_secyear.customer_id
+ and t_s_firstyear.customer_id = t_w_firstyear.customer_id
+ and t_s_firstyear.sale_type = 's'
+ and t_w_firstyear.sale_type = 'w'
+ and t_s_secyear.sale_type = 's'
+ and t_w_secyear.sale_type = 'w'
+ and t_s_firstyear.dyear = 2001
+ and t_s_secyear.dyear = 2001+1
+ and t_w_firstyear.dyear = 2001
+ and t_w_secyear.dyear = 2001+1
+ and t_s_firstyear.year_total > 0
+ and t_w_firstyear.year_total > 0
+ and case when t_w_firstyear.year_total > 0 then 
t_w_secyear.year_total / t_w_firstyear.year_total else null end
+ > case when t_s_firstyear.year_total > 0 then 
t_s_secyear.year_total / t_s_firstyear.year_total else null end
+ order by t_s_secyear.c_preferred_cust_flag
+limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with year_total as (
+ select c_customer_id customer_id
+   ,c_first_name customer_first_name
+   ,c_last_name customer_last_name
+   ,c_preferred_cust_flag
+   ,c_birth_country customer_birth_country
+   ,c_login customer_login
+   ,c_email_address customer_email_address
+   ,d_year dyear
+   ,sum(ss_ext_list_price-ss_ext_discount_amt) year_total
+   ,'s' sale_type
+ from customer
+ ,store_sales
+ ,date_dim
+ where c_customer_sk = ss_customer_sk
+   and ss_sold_date_sk = d_date_sk
+ group by c_customer_id
+ ,c_first_name
+ ,c_last_name
+ ,d_year
+ ,c_preferred_cust_flag
+ ,c_birth_country
+ ,c_login
+ ,c_email_address
+ ,d_year 
+ union all
+ select c_customer_id customer_id
+   ,c_first_name customer_first_name
+   ,c_last_name customer_last_name
+   ,c_preferred_cust_flag
+   ,c_birth_country customer_birth_country
+   ,c_login customer_login
+   ,c_email_address customer_email_address
+   ,d_year dyear
+   ,sum(ws_ext_list_price-ws_ext_discount_amt) year_total
+   ,'w' sale_type
+ from customer
+ ,web_sales
+ ,date_dim
+ where c_customer_sk = ws_bill_customer_sk
+   and ws_sold_date_sk = d_date_sk
+ group by c_customer_id
+ ,c_first_name
+ ,c_last_name
+ ,c_preferred_cust_flag
+ ,c_birth_country
+ ,c_login
+ ,c_email_address
+ ,d_year
+ )
+  select  t_s_secyear.c_preferred_cust_flag
+ from year_total t_s_firstyear
+ ,year_total t_s_secyear
+ ,year_total t_w_firstyear
+ ,year_total t_w_secyear
+ where t_s_secyear.customer_id = t_s_firstyear.customer_id
+ and t_s_firstyear.customer_id = t_w_secyear.customer_id
+ and t_s_firstyear.customer_id = t_w_firstyear.customer_id
+ and t_s_firstyear.sale_type = 's'
+

[30/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query23.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query23.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query23.q.out
new file mode 100644
index 000..8b5a83a
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query23.q.out
@@ -0,0 +1,1110 @@
+Warning: Map Join MAPJOIN[275][bigTable=?] in task 'Stage-1:MAPRED' is a cross 
product
+Warning: Map Join MAPJOIN[276][bigTable=?] in task 'Stage-1:MAPRED' is a cross 
product
+PREHOOK: query: explain
+with frequent_ss_items as 
+ (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date 
solddate,count(*) cnt
+  from store_sales
+  ,date_dim 
+  ,item
+  where ss_sold_date_sk = d_date_sk
+and ss_item_sk = i_item_sk 
+and d_year in (1999,1999+1,1999+2,1999+3)
+  group by substr(i_item_desc,1,30),i_item_sk,d_date
+  having count(*) >4),
+ max_store_sales as
+ (select max(csales) tpcds_cmax 
+  from (select c_customer_sk,sum(ss_quantity*ss_sales_price) csales
+from store_sales
+,customer
+,date_dim 
+where ss_customer_sk = c_customer_sk
+ and ss_sold_date_sk = d_date_sk
+ and d_year in (1999,1999+1,1999+2,1999+3) 
+group by c_customer_sk) x),
+ best_ss_customer as
+ (select c_customer_sk,sum(ss_quantity*ss_sales_price) ssales
+  from store_sales
+  ,customer
+  where ss_customer_sk = c_customer_sk
+  group by c_customer_sk
+  having sum(ss_quantity*ss_sales_price) > (95/100.0) * (select
+  *
+from
+ max_store_sales))
+  select  sum(sales)
+ from ((select cs_quantity*cs_list_price sales
+   from catalog_sales
+   ,date_dim 
+   where d_year = 1999 
+ and d_moy = 1 
+ and cs_sold_date_sk = d_date_sk 
+ and cs_item_sk in (select item_sk from frequent_ss_items)
+ and cs_bill_customer_sk in (select c_customer_sk from 
best_ss_customer))
+  union all
+  (select ws_quantity*ws_list_price sales
+   from web_sales 
+   ,date_dim 
+   where d_year = 1999 
+ and d_moy = 1 
+ and ws_sold_date_sk = d_date_sk 
+ and ws_item_sk in (select item_sk from frequent_ss_items)
+ and ws_bill_customer_sk in (select c_customer_sk from 
best_ss_customer))) y
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with frequent_ss_items as 
+ (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date 
solddate,count(*) cnt
+  from store_sales
+  ,date_dim 
+  ,item
+  where ss_sold_date_sk = d_date_sk
+and ss_item_sk = i_item_sk 
+and d_year in (1999,1999+1,1999+2,1999+3)
+  group by substr(i_item_desc,1,30),i_item_sk,d_date
+  having count(*) >4),
+ max_store_sales as
+ (select max(csales) tpcds_cmax 
+  from (select c_customer_sk,sum(ss_quantity*ss_sales_price) csales
+from store_sales
+,customer
+,date_dim 
+where ss_customer_sk = c_customer_sk
+ and ss_sold_date_sk = d_date_sk
+ and d_year in (1999,1999+1,1999+2,1999+3) 
+group by c_customer_sk) x),
+ best_ss_customer as
+ (select c_customer_sk,sum(ss_quantity*ss_sales_price) ssales
+  from store_sales
+  ,customer
+  where ss_customer_sk = c_customer_sk
+  group by c_customer_sk
+  having sum(ss_quantity*ss_sales_price) > (95/100.0) * (select
+  *
+from
+ max_store_sales))
+  select  sum(sales)
+ from ((select cs_quantity*cs_list_price sales
+   from catalog_sales
+   ,date_dim 
+   where d_year = 1999 
+ and d_moy = 1 
+ and cs_sold_date_sk = d_date_sk 
+ and cs_item_sk in (select item_sk from frequent_ss_items)
+ and cs_bill_customer_sk in (select c_customer_sk from 
best_ss_customer))
+  union all
+  (select ws_quantity*ws_list_price sales
+   from web_sales 
+   ,date_dim 
+   where d_year = 1999 
+ and d_moy = 1 
+ and ws_sold_date_sk = d_date_sk 
+ and ws_item_sk in (select item_sk from frequent_ss_items)
+ and ws_bill_customer_sk in (select c_customer_sk from 
best_ss_customer))) y
+ limit 100
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+  Stage-2 is a root stage
+  Stage-3 depends on stages: Stage-2
+  Stage-1 depends on stages: Stage-3
+  Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+  Stage: Stage-2
+Spark
+  Edges:
+Reducer 14 <- Map 13 (PARTITION-LEVEL SORT, 398), Map 18 
(PARTITION-LEVEL SORT, 398)
+Reducer 15 <- Map 19 (PARTITION-LEVEL SORT, 975), Reducer 14 
(PARTITION-LEVEL SORT, 975)
+Reducer 16 <- Reducer 15 (GROUP, 481)
+Reducer 17 <- Reducer 16 (GROUP, 1)
+Reducer 21 <- Map 20 (PARTITION-LEVEL SORT, 398), Map 25 
(PARTITION-LEVEL SORT, 398)
+Reducer 22 <- Map 26 (PARTITION-LEVEL SORT, 975), Reducer 21 
(PARTITION-LEVEL SORT, 975)
+Reducer 2

[25/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query36.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query36.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query36.q.out
new file mode 100644
index 000..c072728
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query36.q.out
@@ -0,0 +1,282 @@
+PREHOOK: query: explain
+select  
+sum(ss_net_profit)/sum(ss_ext_sales_price) as gross_margin
+   ,i_category
+   ,i_class
+   ,grouping(i_category)+grouping(i_class) as lochierarchy
+   ,rank() over (
+   partition by grouping(i_category)+grouping(i_class),
+   case when grouping(i_class) = 0 then i_category end 
+   order by sum(ss_net_profit)/sum(ss_ext_sales_price) asc) as 
rank_within_parent
+ from
+store_sales
+   ,date_dim   d1
+   ,item
+   ,store
+ where
+d1.d_year = 1999 
+ and d1.d_date_sk = ss_sold_date_sk
+ and i_item_sk  = ss_item_sk 
+ and s_store_sk  = ss_store_sk
+ and s_state in ('SD','FL','MI','LA',
+ 'MO','SC','AL','GA')
+ group by rollup(i_category,i_class)
+ order by
+   lochierarchy desc
+  ,case when lochierarchy = 0 then i_category end
+  ,rank_within_parent
+  limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select  
+sum(ss_net_profit)/sum(ss_ext_sales_price) as gross_margin
+   ,i_category
+   ,i_class
+   ,grouping(i_category)+grouping(i_class) as lochierarchy
+   ,rank() over (
+   partition by grouping(i_category)+grouping(i_class),
+   case when grouping(i_class) = 0 then i_category end 
+   order by sum(ss_net_profit)/sum(ss_ext_sales_price) asc) as 
rank_within_parent
+ from
+store_sales
+   ,date_dim   d1
+   ,item
+   ,store
+ where
+d1.d_year = 1999 
+ and d1.d_date_sk = ss_sold_date_sk
+ and i_item_sk  = ss_item_sk 
+ and s_store_sk  = ss_store_sk
+ and s_state in ('SD','FL','MI','LA',
+ 'MO','SC','AL','GA')
+ group by rollup(i_category,i_class)
+ order by
+   lochierarchy desc
+  ,case when lochierarchy = 0 then i_category end
+  ,rank_within_parent
+  limit 100
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+  Stage-2 is a root stage
+  Stage-1 depends on stages: Stage-2
+  Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+  Stage: Stage-2
+Spark
+ A masked pattern was here 
+  Vertices:
+Map 8 
+Map Operator Tree:
+TableScan
+  alias: store
+  Statistics: Num rows: 1704 Data size: 3256276 Basic stats: 
COMPLETE Column stats: NONE
+  Filter Operator
+predicate: ((s_state) IN ('SD', 'FL', 'MI', 'LA', 'MO', 
'SC', 'AL', 'GA') and s_store_sk is not null) (type: boolean)
+Statistics: Num rows: 852 Data size: 1628138 Basic stats: 
COMPLETE Column stats: NONE
+Select Operator
+  expressions: s_store_sk (type: int)
+  outputColumnNames: _col0
+  Statistics: Num rows: 852 Data size: 1628138 Basic 
stats: COMPLETE Column stats: NONE
+  Spark HashTable Sink Operator
+keys:
+  0 _col2 (type: int)
+  1 _col0 (type: int)
+Local Work:
+  Map Reduce Local Work
+
+  Stage: Stage-1
+Spark
+  Edges:
+Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 398), Map 7 (PARTITION-LEVEL 
SORT, 398)
+Reducer 3 <- Map 9 (PARTITION-LEVEL SORT, 486), Reducer 2 
(PARTITION-LEVEL SORT, 486)
+Reducer 4 <- Reducer 3 (GROUP, 1009)
+Reducer 5 <- Reducer 4 (PARTITION-LEVEL SORT, 793)
+Reducer 6 <- Reducer 5 (SORT, 1)
+ A masked pattern was here 
+  Vertices:
+Map 1 
+Map Operator Tree:
+TableScan
+  alias: store_sales
+  Statistics: Num rows: 575995635 Data size: 50814502088 Basic 
stats: COMPLETE Column stats: NONE
+  Filter Operator
+predicate: (ss_item_sk is not null and ss_sold_date_sk is 
not null and ss_store_sk is not null) (type: boolean)
+Statistics: Num rows: 575995635 Data size: 50814502088 
Basic stats: COMPLETE Column stats: NONE
+Select Operator
+  expressions: ss_sold_date_sk (type: int), ss_item_sk 
(type: int), ss_store_sk (type: int), ss_ext_sales_price (type: decimal(7,2)), 
ss_net_profit (type: decimal(7,2))
+  outputColumnNames: _col0, _col1, _col2, _col3, _col4
+  Statistics: Num rows: 575995635 Data size: 50814502088 
Basic stats: COMPLETE Column stats: NONE
+  Reduce Output Operator
+key expressions: _col0 (type: int)
+sort order: +
+Map-reduce partition c

[35/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)


Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/bd371246
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/bd371246
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/bd371246

Branch: refs/heads/master
Commit: bd371246e2cb15f076574925b02bb31a43ac94e7
Parents: 365c031
Author: Sahil Takiar 
Authored: Thu Sep 28 10:01:10 2017 -0700
Committer: Sahil Takiar 
Committed: Thu Sep 28 11:11:42 2017 -0700

--
 data/conf/perf-reg/spark/hive-site.xml  |  268 ++
 .../hadoop/hive/cli/TestSparkPerfCliDriver.java |   62 +
 .../hadoop/hive/cli/TestPerfCliDriver.java  |   62 -
 .../hadoop/hive/cli/TestTezPerfCliDriver.java   |   62 +
 .../test/resources/testconfiguration.properties |2 +
 .../hadoop/hive/cli/control/CliConfigs.java |   32 +-
 .../physical/SparkCrossProductCheck.java|2 +-
 .../apache/hadoop/hive/ql/plan/SparkWork.java   |4 +-
 .../clientpositive/perf/spark/query1.q.out  |  340 ++
 .../clientpositive/perf/spark/query10.q.out |  533 +++
 .../clientpositive/perf/spark/query11.q.out |  683 
 .../clientpositive/perf/spark/query12.q.out |  249 ++
 .../clientpositive/perf/spark/query13.q.out |  339 ++
 .../clientpositive/perf/spark/query14.q.out | 3237 ++
 .../clientpositive/perf/spark/query15.q.out |  223 ++
 .../clientpositive/perf/spark/query16.q.out |  363 ++
 .../clientpositive/perf/spark/query17.q.out |  409 +++
 .../clientpositive/perf/spark/query18.q.out |  360 ++
 .../clientpositive/perf/spark/query19.q.out |  306 ++
 .../clientpositive/perf/spark/query2.q.out  |  429 +++
 .../clientpositive/perf/spark/query20.q.out |  241 ++
 .../clientpositive/perf/spark/query21.q.out |  245 ++
 .../clientpositive/perf/spark/query22.q.out |  218 ++
 .../clientpositive/perf/spark/query23.q.out | 1110 ++
 .../clientpositive/perf/spark/query24.q.out |  610 
 .../clientpositive/perf/spark/query25.q.out |  415 +++
 .../clientpositive/perf/spark/query26.q.out |  253 ++
 .../clientpositive/perf/spark/query27.q.out |  265 ++
 .../clientpositive/perf/spark/query28.q.out |  517 +++
 .../clientpositive/perf/spark/query29.q.out |  415 +++
 .../clientpositive/perf/spark/query3.q.out  |  183 +
 .../clientpositive/perf/spark/query30.q.out |  424 +++
 .../clientpositive/perf/spark/query31.q.out |  815 +
 .../clientpositive/perf/spark/query32.q.out |  284 ++
 .../clientpositive/perf/spark/query33.q.out |  683 
 .../clientpositive/perf/spark/query34.q.out |  273 ++
 .../clientpositive/perf/spark/query35.q.out |  524 +++
 .../clientpositive/perf/spark/query36.q.out |  282 ++
 .../clientpositive/perf/spark/query37.q.out |  192 ++
 .../clientpositive/perf/spark/query38.q.out |  458 +++
 .../clientpositive/perf/spark/query39.q.out |  473 +++
 .../clientpositive/perf/spark/query4.q.out  |  986 ++
 .../clientpositive/perf/spark/query40.q.out |  274 ++
 .../clientpositive/perf/spark/query42.q.out |  191 ++
 .../clientpositive/perf/spark/query43.q.out |  184 +
 .../clientpositive/perf/spark/query44.q.out |  486 +++
 .../clientpositive/perf/spark/query45.q.out |  374 ++
 .../clientpositive/perf/spark/query46.q.out |  355 ++
 .../clientpositive/perf/spark/query47.q.out |  771 +
 .../clientpositive/perf/spark/query48.q.out |  329 ++
 .../clientpositive/perf/spark/query49.q.out |  876 +
 .../clientpositive/perf/spark/query5.q.out  |  740 
 .../clientpositive/perf/spark/query50.q.out |  333 ++
 .../clientpositive/perf/spark/query51.q.out |  383 +++
 .../clientpositive/perf/spark/query52.q.out |  188 +
 .../clientpositive/perf/spark/query53.q.out |  267 ++
 .../clientpositive/perf/spark/query54.q.out |  742 
 .../clientpositive/perf/spark/query55.q.out |  168 +
 .../clientpositive/perf/spark/query56.q.out |  669 
 .../clientpositive/perf/spark/query57.q.out |  765 +
 .../clientpositive/perf/spark/query58.q.out |  944 +
 .../clientpositive/perf/spark/query59.q.out |  445 +++
 .../clientpositive/perf/spark/query6.q.out  |  459 +++
 .../clientpositive/perf/spark/query60.q.out |  691 
 .../clientpositive/perf/spark/query61.q.out |  586 
 .../clientpositive/perf/spark/query63.q.out |  269 ++
 .../clientpositive/perf/spark/query65.q.out |  333 ++
 .../clientpositive/perf/spark/query66.q.out |  873 +
 .../clientpositive/perf/spark/query67.q.out |  315 ++
 .../clientpositive/perf/spark/query68.q.out |  363 ++
 .../clientpositive/perf/spark/query69.q.out |  516 +++
 .../clientpositive/perf/spark/query7.q.out  |  253 ++
 .../clientpositive/perf/spark/query70.q.out |  429

[26/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query33.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query33.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query33.q.out
new file mode 100644
index 000..dcf2fef
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query33.q.out
@@ -0,0 +1,683 @@
+PREHOOK: query: explain
+with ss as (
+ select
+  i_manufact_id,sum(ss_ext_sales_price) total_sales
+ from
+   store_sales,
+   date_dim,
+ customer_address,
+ item
+ where
+ i_manufact_id in (select
+  i_manufact_id
+from
+ item
+where i_category in ('Books'))
+ and ss_item_sk  = i_item_sk
+ and ss_sold_date_sk = d_date_sk
+ and d_year  = 1999
+ and d_moy   = 3
+ and ss_addr_sk  = ca_address_sk
+ and ca_gmt_offset   = -6 
+ group by i_manufact_id),
+ cs as (
+ select
+  i_manufact_id,sum(cs_ext_sales_price) total_sales
+ from
+   catalog_sales,
+   date_dim,
+ customer_address,
+ item
+ where
+ i_manufact_id   in (select
+  i_manufact_id
+from
+ item
+where i_category in ('Books'))
+ and cs_item_sk  = i_item_sk
+ and cs_sold_date_sk = d_date_sk
+ and d_year  = 1999
+ and d_moy   = 3
+ and cs_bill_addr_sk = ca_address_sk
+ and ca_gmt_offset   = -6 
+ group by i_manufact_id),
+ ws as (
+ select
+  i_manufact_id,sum(ws_ext_sales_price) total_sales
+ from
+   web_sales,
+   date_dim,
+ customer_address,
+ item
+ where
+ i_manufact_id   in (select
+  i_manufact_id
+from
+ item
+where i_category in ('Books'))
+ and ws_item_sk  = i_item_sk
+ and ws_sold_date_sk = d_date_sk
+ and d_year  = 1999
+ and d_moy   = 3
+ and ws_bill_addr_sk = ca_address_sk
+ and ca_gmt_offset   = -6
+ group by i_manufact_id)
+  select  i_manufact_id ,sum(total_sales) total_sales
+ from  (select * from ss 
+union all
+select * from cs 
+union all
+select * from ws) tmp1
+ group by i_manufact_id
+ order by total_sales
+limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with ss as (
+ select
+  i_manufact_id,sum(ss_ext_sales_price) total_sales
+ from
+   store_sales,
+   date_dim,
+ customer_address,
+ item
+ where
+ i_manufact_id in (select
+  i_manufact_id
+from
+ item
+where i_category in ('Books'))
+ and ss_item_sk  = i_item_sk
+ and ss_sold_date_sk = d_date_sk
+ and d_year  = 1999
+ and d_moy   = 3
+ and ss_addr_sk  = ca_address_sk
+ and ca_gmt_offset   = -6 
+ group by i_manufact_id),
+ cs as (
+ select
+  i_manufact_id,sum(cs_ext_sales_price) total_sales
+ from
+   catalog_sales,
+   date_dim,
+ customer_address,
+ item
+ where
+ i_manufact_id   in (select
+  i_manufact_id
+from
+ item
+where i_category in ('Books'))
+ and cs_item_sk  = i_item_sk
+ and cs_sold_date_sk = d_date_sk
+ and d_year  = 1999
+ and d_moy   = 3
+ and cs_bill_addr_sk = ca_address_sk
+ and ca_gmt_offset   = -6 
+ group by i_manufact_id),
+ ws as (
+ select
+  i_manufact_id,sum(ws_ext_sales_price) total_sales
+ from
+   web_sales,
+   date_dim,
+ customer_address,
+ item
+ where
+ i_manufact_id   in (select
+  i_manufact_id
+from
+ item
+where i_category in ('Books'))
+ and ws_item_sk  = i_item_sk
+ and ws_sold_date_sk = d_date_sk
+ and d_year  = 1999
+ and d_moy   = 3
+ and ws_bill_addr_sk = ca_address_sk
+ and ca_gmt_offset   = -6
+ group by i_manufact_id)
+  select  i_manufact_id ,sum(total_sales) total_sales
+ from  (select * from ss 
+union all
+select * from cs 
+union all
+select * from ws) tmp1
+ group by i_manufact_id
+ order by total_sales
+limit 100
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+  Stage-1 is a root stage
+  Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+  Stage: Stage-1
+Spark
+  Edges:
+Reducer 10 <- Map 12 (PARTITION-LEVEL SORT, 398), Map 9 
(PARTITION-LEVEL SORT, 398)
+Reducer 11 <- Map 13 (PARTITION-LEVEL SORT, 596), Reducer 10 
(PARTITION-LEVEL SORT, 596)
+Reducer 15 <- Map 1 (PARTITION-LEVEL SORT, 7), Reducer 19 
(PARTITION-LEVEL SORT, 7)
+Reducer 16 <- Reducer 15 (PARTITION-LEVEL SORT, 375), Reducer 22 
(PARTITION-LEVEL SORT

[16/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query6.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query6.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query6.q.out
new file mode 100644
index 000..126cf9e
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query6.q.out
@@ -0,0 +1,459 @@
+Warning: Map Join MAPJOIN[85][bigTable=?] in task 'Stage-1:MAPRED' is a cross 
product
+PREHOOK: query: explain
+select  a.ca_state state, count(*) cnt
+ from customer_address a
+ ,customer c
+ ,store_sales s
+ ,date_dim d
+ ,item i
+ where   a.ca_address_sk = c.c_current_addr_sk
+   and c.c_customer_sk = s.ss_customer_sk
+   and s.ss_sold_date_sk = d.d_date_sk
+   and s.ss_item_sk = i.i_item_sk
+   and d.d_month_seq = 
+(select distinct (d_month_seq)
+ from date_dim
+   where d_year = 2000
+   and d_moy = 2 )
+   and i.i_current_price > 1.2 * 
+ (select avg(j.i_current_price) 
+from item j 
+where j.i_category = i.i_category)
+ group by a.ca_state
+ having count(*) >= 10
+ order by cnt 
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select  a.ca_state state, count(*) cnt
+ from customer_address a
+ ,customer c
+ ,store_sales s
+ ,date_dim d
+ ,item i
+ where   a.ca_address_sk = c.c_current_addr_sk
+   and c.c_customer_sk = s.ss_customer_sk
+   and s.ss_sold_date_sk = d.d_date_sk
+   and s.ss_item_sk = i.i_item_sk
+   and d.d_month_seq = 
+(select distinct (d_month_seq)
+ from date_dim
+   where d_year = 2000
+   and d_moy = 2 )
+   and i.i_current_price > 1.2 * 
+ (select avg(j.i_current_price) 
+from item j 
+where j.i_category = i.i_category)
+ group by a.ca_state
+ having count(*) >= 10
+ order by cnt 
+ limit 100
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+  Stage-2 is a root stage
+  Stage-1 depends on stages: Stage-2
+  Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+  Stage: Stage-2
+Spark
+  Edges:
+Reducer 18 <- Map 17 (GROUP, 2)
+Reducer 19 <- Reducer 18 (GROUP, 1)
+ A masked pattern was here 
+  Vertices:
+Map 17 
+Map Operator Tree:
+TableScan
+  alias: date_dim
+  Statistics: Num rows: 73049 Data size: 81741831 Basic stats: 
COMPLETE Column stats: NONE
+  Filter Operator
+predicate: ((d_moy = 2) and (d_year = 2000)) (type: 
boolean)
+Statistics: Num rows: 18262 Data size: 20435178 Basic 
stats: COMPLETE Column stats: NONE
+Select Operator
+  expressions: d_month_seq (type: int)
+  outputColumnNames: d_month_seq
+  Statistics: Num rows: 18262 Data size: 20435178 Basic 
stats: COMPLETE Column stats: NONE
+  Group By Operator
+keys: d_month_seq (type: int)
+mode: hash
+outputColumnNames: _col0
+Statistics: Num rows: 18262 Data size: 20435178 Basic 
stats: COMPLETE Column stats: NONE
+Reduce Output Operator
+  key expressions: _col0 (type: int)
+  sort order: +
+  Map-reduce partition columns: _col0 (type: int)
+  Statistics: Num rows: 18262 Data size: 20435178 
Basic stats: COMPLETE Column stats: NONE
+Reducer 18 
+Reduce Operator Tree:
+  Group By Operator
+keys: KEY._col0 (type: int)
+mode: mergepartial
+outputColumnNames: _col0
+Statistics: Num rows: 9131 Data size: 10217589 Basic stats: 
COMPLETE Column stats: NONE
+Select Operator
+  Statistics: Num rows: 9131 Data size: 10217589 Basic stats: 
COMPLETE Column stats: NONE
+  Group By Operator
+aggregations: count()
+mode: hash
+outputColumnNames: _col0
+Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE 
Column stats: NONE
+Reduce Output Operator
+  sort order: 
+  Statistics: Num rows: 1 Data size: 8 Basic stats: 
COMPLETE Column stats: NONE
+  value expressions: _col0 (type: bigint)
+Reducer 19 
+Local Work:
+  Map Reduce Local Work
+Reduce Operator Tree:
+  Group By Operator
+aggregations: count(VALUE._col0)
+mode: mergepartial
+outputColumnNames:

[33/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query14.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query14.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query14.q.out
new file mode 100644
index 000..f52e960
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query14.q.out
@@ -0,0 +1,3237 @@
+Warning: Map Join MAPJOIN[650][bigTable=?] in task 'Stage-1:MAPRED' is a cross 
product
+Warning: Map Join MAPJOIN[672][bigTable=?] in task 'Stage-1:MAPRED' is a cross 
product
+Warning: Map Join MAPJOIN[661][bigTable=?] in task 'Stage-1:MAPRED' is a cross 
product
+PREHOOK: query: explain
+with  cross_items as
+ (select i_item_sk ss_item_sk
+ from item,
+ (select iss.i_brand_id brand_id
+ ,iss.i_class_id class_id
+ ,iss.i_category_id category_id
+ from store_sales
+ ,item iss
+ ,date_dim d1
+ where ss_item_sk = iss.i_item_sk
+   and ss_sold_date_sk = d1.d_date_sk
+   and d1.d_year between 1999 AND 1999 + 2
+ intersect 
+ select ics.i_brand_id
+ ,ics.i_class_id
+ ,ics.i_category_id
+ from catalog_sales
+ ,item ics
+ ,date_dim d2
+ where cs_item_sk = ics.i_item_sk
+   and cs_sold_date_sk = d2.d_date_sk
+   and d2.d_year between 1999 AND 1999 + 2
+ intersect
+ select iws.i_brand_id
+ ,iws.i_class_id
+ ,iws.i_category_id
+ from web_sales
+ ,item iws
+ ,date_dim d3
+ where ws_item_sk = iws.i_item_sk
+   and ws_sold_date_sk = d3.d_date_sk
+   and d3.d_year between 1999 AND 1999 + 2) x
+ where i_brand_id = brand_id
+  and i_class_id = class_id
+  and i_category_id = category_id
+),
+ avg_sales as
+ (select avg(quantity*list_price) average_sales
+  from (select ss_quantity quantity
+ ,ss_list_price list_price
+   from store_sales
+   ,date_dim
+   where ss_sold_date_sk = d_date_sk
+ and d_year between 1999 and 2001 
+   union all 
+   select cs_quantity quantity 
+ ,cs_list_price list_price
+   from catalog_sales
+   ,date_dim
+   where cs_sold_date_sk = d_date_sk
+ and d_year between 1998 and 1998 + 2 
+   union all
+   select ws_quantity quantity
+ ,ws_list_price list_price
+   from web_sales
+   ,date_dim
+   where ws_sold_date_sk = d_date_sk
+ and d_year between 1998 and 1998 + 2) x)
+  select  channel, i_brand_id,i_class_id,i_category_id,sum(sales), 
sum(number_sales)
+ from(
+   select 'store' channel, i_brand_id,i_class_id
+ ,i_category_id,sum(ss_quantity*ss_list_price) sales
+ , count(*) number_sales
+   from store_sales
+   ,item
+   ,date_dim
+   where ss_item_sk in (select ss_item_sk from cross_items)
+ and ss_item_sk = i_item_sk
+ and ss_sold_date_sk = d_date_sk
+ and d_year = 1998+2 
+ and d_moy = 11
+   group by i_brand_id,i_class_id,i_category_id
+   having sum(ss_quantity*ss_list_price) > (select average_sales from 
avg_sales)
+   union all
+   select 'catalog' channel, i_brand_id,i_class_id,i_category_id, 
sum(cs_quantity*cs_list_price) sales, count(*) number_sales
+   from catalog_sales
+   ,item
+   ,date_dim
+   where cs_item_sk in (select ss_item_sk from cross_items)
+ and cs_item_sk = i_item_sk
+ and cs_sold_date_sk = d_date_sk
+ and d_year = 1998+2 
+ and d_moy = 11
+   group by i_brand_id,i_class_id,i_category_id
+   having sum(cs_quantity*cs_list_price) > (select average_sales from 
avg_sales)
+   union all
+   select 'web' channel, i_brand_id,i_class_id,i_category_id, 
sum(ws_quantity*ws_list_price) sales , count(*) number_sales
+   from web_sales
+   ,item
+   ,date_dim
+   where ws_item_sk in (select ss_item_sk from cross_items)
+ and ws_item_sk = i_item_sk
+ and ws_sold_date_sk = d_date_sk
+ and d_year = 1998+2
+ and d_moy = 11
+   group by i_brand_id,i_class_id,i_category_id
+   having sum(ws_quantity*ws_list_price) > (select average_sales from 
avg_sales)
+ ) y
+ group by rollup (channel, i_brand_id,i_class_id,i_category_id)
+ order by channel,i_brand_id,i_class_id,i_category_id
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with  cross_items as
+ (select i_item_sk ss_item_sk
+ from item,
+ (select iss.i_brand_id brand_id
+ ,iss.i_class_id class_id
+ ,iss.i_category_id category_id
+ from store_sales
+ ,item iss
+ ,date_dim d1
+ where ss_item_sk = iss.i_item_sk
+   and ss_sold_date_sk = d1.d_date_sk
+   and d1.d_year between 1999 AND 1999 + 2
+ intersect 
+ select ics.i_brand_id
+ ,ics.i_class_id
+ ,ics.i_category_id
+ from catalog_sales
+ ,item ics
+ ,date_dim d2
+ where cs_item_sk = ics.i_item_sk
+   and cs_sold_date_sk = d2.d_date_sk
+   and d2.d_year between 1999 AND

[24/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query4.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query4.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query4.q.out
new file mode 100644
index 000..71154ff
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query4.q.out
@@ -0,0 +1,986 @@
+PREHOOK: query: explain
+with year_total as (
+ select c_customer_id customer_id
+   ,c_first_name customer_first_name
+   ,c_last_name customer_last_name
+   ,c_preferred_cust_flag customer_preferred_cust_flag
+   ,c_birth_country customer_birth_country
+   ,c_login customer_login
+   ,c_email_address customer_email_address
+   ,d_year dyear
+   
,sum(((ss_ext_list_price-ss_ext_wholesale_cost-ss_ext_discount_amt)+ss_ext_sales_price)/2)
 year_total
+   ,'s' sale_type
+ from customer
+ ,store_sales
+ ,date_dim
+ where c_customer_sk = ss_customer_sk
+   and ss_sold_date_sk = d_date_sk
+ group by c_customer_id
+ ,c_first_name
+ ,c_last_name
+ ,c_preferred_cust_flag
+ ,c_birth_country
+ ,c_login
+ ,c_email_address
+ ,d_year
+ union all
+ select c_customer_id customer_id
+   ,c_first_name customer_first_name
+   ,c_last_name customer_last_name
+   ,c_preferred_cust_flag customer_preferred_cust_flag
+   ,c_birth_country customer_birth_country
+   ,c_login customer_login
+   ,c_email_address customer_email_address
+   ,d_year dyear
+   
,sumcs_ext_list_price-cs_ext_wholesale_cost-cs_ext_discount_amt)+cs_ext_sales_price)/2)
 ) year_total
+   ,'c' sale_type
+ from customer
+ ,catalog_sales
+ ,date_dim
+ where c_customer_sk = cs_bill_customer_sk
+   and cs_sold_date_sk = d_date_sk
+ group by c_customer_id
+ ,c_first_name
+ ,c_last_name
+ ,c_preferred_cust_flag
+ ,c_birth_country
+ ,c_login
+ ,c_email_address
+ ,d_year
+union all
+ select c_customer_id customer_id
+   ,c_first_name customer_first_name
+   ,c_last_name customer_last_name
+   ,c_preferred_cust_flag customer_preferred_cust_flag
+   ,c_birth_country customer_birth_country
+   ,c_login customer_login
+   ,c_email_address customer_email_address
+   ,d_year dyear
+   
,sumws_ext_list_price-ws_ext_wholesale_cost-ws_ext_discount_amt)+ws_ext_sales_price)/2)
 ) year_total
+   ,'w' sale_type
+ from customer
+ ,web_sales
+ ,date_dim
+ where c_customer_sk = ws_bill_customer_sk
+   and ws_sold_date_sk = d_date_sk
+ group by c_customer_id
+ ,c_first_name
+ ,c_last_name
+ ,c_preferred_cust_flag
+ ,c_birth_country
+ ,c_login
+ ,c_email_address
+ ,d_year
+ )
+  select  t_s_secyear.customer_preferred_cust_flag
+ from year_total t_s_firstyear
+ ,year_total t_s_secyear
+ ,year_total t_c_firstyear
+ ,year_total t_c_secyear
+ ,year_total t_w_firstyear
+ ,year_total t_w_secyear
+ where t_s_secyear.customer_id = t_s_firstyear.customer_id
+   and t_s_firstyear.customer_id = t_c_secyear.customer_id
+   and t_s_firstyear.customer_id = t_c_firstyear.customer_id
+   and t_s_firstyear.customer_id = t_w_firstyear.customer_id
+   and t_s_firstyear.customer_id = t_w_secyear.customer_id
+   and t_s_firstyear.sale_type = 's'
+   and t_c_firstyear.sale_type = 'c'
+   and t_w_firstyear.sale_type = 'w'
+   and t_s_secyear.sale_type = 's'
+   and t_c_secyear.sale_type = 'c'
+   and t_w_secyear.sale_type = 'w'
+   and t_s_firstyear.dyear =  2001
+   and t_s_secyear.dyear = 2001+1
+   and t_c_firstyear.dyear =  2001
+   and t_c_secyear.dyear =  2001+1
+   and t_w_firstyear.dyear = 2001
+   and t_w_secyear.dyear = 2001+1
+   and t_s_firstyear.year_total > 0
+   and t_c_firstyear.year_total > 0
+   and t_w_firstyear.year_total > 0
+   and case when t_c_firstyear.year_total > 0 then t_c_secyear.year_total / 
t_c_firstyear.year_total else null end
+   > case when t_s_firstyear.year_total > 0 then 
t_s_secyear.year_total / t_s_firstyear.year_total else null end
+   and case when t_c_firstyear.year_total > 0 then t_c_secyear.year_total / 
t_c_firstyear.year_total else null end
+   > case when t_w_firstyear.year_total > 0 then 
t_w_secyear.year_total / t_w_firstyear.year_total else null end
+ order by t_s_secyear.customer_preferred_cust_flag
+limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with year_total as (
+ select c_customer_id customer_id
+   ,c_first_name customer_first_name
+   ,c_last_name customer_last_name
+   ,c_preferred_cust_flag customer_preferred_cust_flag
+   ,c_birth_country customer_birth_country
+   ,c_login customer_login
+   ,c_email_address customer_email_address
+   ,d_year dyear
+   
,sum(((ss_ext_list_price-ss_ext_wholesale_cost-ss

[10/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query75.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query75.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query75.q.out
new file mode 100644
index 000..654cdf9
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query75.q.out
@@ -0,0 +1,898 @@
+PREHOOK: query: explain
+WITH all_sales AS (
+ SELECT d_year
+   ,i_brand_id
+   ,i_class_id
+   ,i_category_id
+   ,i_manufact_id
+   ,SUM(sales_cnt) AS sales_cnt
+   ,SUM(sales_amt) AS sales_amt
+ FROM (SELECT d_year
+ ,i_brand_id
+ ,i_class_id
+ ,i_category_id
+ ,i_manufact_id
+ ,cs_quantity - COALESCE(cr_return_quantity,0) AS sales_cnt
+ ,cs_ext_sales_price - COALESCE(cr_return_amount,0.0) AS sales_amt
+   FROM catalog_sales JOIN item ON i_item_sk=cs_item_sk
+  JOIN date_dim ON d_date_sk=cs_sold_date_sk
+  LEFT JOIN catalog_returns ON 
(cs_order_number=cr_order_number 
+AND cs_item_sk=cr_item_sk)
+   WHERE i_category='Sports'
+   UNION
+   SELECT d_year
+ ,i_brand_id
+ ,i_class_id
+ ,i_category_id
+ ,i_manufact_id
+ ,ss_quantity - COALESCE(sr_return_quantity,0) AS sales_cnt
+ ,ss_ext_sales_price - COALESCE(sr_return_amt,0.0) AS sales_amt
+   FROM store_sales JOIN item ON i_item_sk=ss_item_sk
+JOIN date_dim ON d_date_sk=ss_sold_date_sk
+LEFT JOIN store_returns ON 
(ss_ticket_number=sr_ticket_number 
+AND ss_item_sk=sr_item_sk)
+   WHERE i_category='Sports'
+   UNION
+   SELECT d_year
+ ,i_brand_id
+ ,i_class_id
+ ,i_category_id
+ ,i_manufact_id
+ ,ws_quantity - COALESCE(wr_return_quantity,0) AS sales_cnt
+ ,ws_ext_sales_price - COALESCE(wr_return_amt,0.0) AS sales_amt
+   FROM web_sales JOIN item ON i_item_sk=ws_item_sk
+  JOIN date_dim ON d_date_sk=ws_sold_date_sk
+  LEFT JOIN web_returns ON 
(ws_order_number=wr_order_number 
+AND ws_item_sk=wr_item_sk)
+   WHERE i_category='Sports') sales_detail
+ GROUP BY d_year, i_brand_id, i_class_id, i_category_id, i_manufact_id)
+ SELECT  prev_yr.d_year AS prev_year
+  ,curr_yr.d_year AS year
+  ,curr_yr.i_brand_id
+  ,curr_yr.i_class_id
+  ,curr_yr.i_category_id
+  ,curr_yr.i_manufact_id
+  ,prev_yr.sales_cnt AS prev_yr_cnt
+  ,curr_yr.sales_cnt AS curr_yr_cnt
+  ,curr_yr.sales_cnt-prev_yr.sales_cnt AS 
sales_cnt_diff
+  ,curr_yr.sales_amt-prev_yr.sales_amt AS 
sales_amt_diff
+ FROM all_sales curr_yr, all_sales prev_yr
+ WHERE curr_yr.i_brand_id=prev_yr.i_brand_id
+   AND curr_yr.i_class_id=prev_yr.i_class_id
+   AND curr_yr.i_category_id=prev_yr.i_category_id
+   AND curr_yr.i_manufact_id=prev_yr.i_manufact_id
+   AND curr_yr.d_year=2002
+   AND prev_yr.d_year=2002-1
+   AND CAST(curr_yr.sales_cnt AS DECIMAL(17,2))/CAST(prev_yr.sales_cnt AS 
DECIMAL(17,2))<0.9
+ ORDER BY sales_cnt_diff
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+WITH all_sales AS (
+ SELECT d_year
+   ,i_brand_id
+   ,i_class_id
+   ,i_category_id
+   ,i_manufact_id
+   ,SUM(sales_cnt) AS sales_cnt
+   ,SUM(sales_amt) AS sales_amt
+ FROM (SELECT d_year
+ ,i_brand_id
+ ,i_class_id
+ ,i_category_id
+ ,i_manufact_id
+ ,cs_quantity - COALESCE(cr_return_quantity,0) AS sales_cnt
+ ,cs_ext_sales_price - COALESCE(cr_return_amount,0.0) AS sales_amt
+   FROM catalog_sales JOIN item ON i_item_sk=cs_item_sk
+  JOIN date_dim ON d_date_sk=cs_sold_date_sk
+  LEFT JOIN catalog_returns ON 
(cs_order_number=cr_order_number 
+AND cs_item_sk=cr_item_sk)
+   WHERE i_category='Sports'
+   UNION
+   SELECT d_year
+ ,i_brand_id
+ ,i_class_id
+ ,i_category_id
+ ,i_manufact_id
+ ,ss_quantity - COALESCE(sr_return_quantity,0) AS sales_cnt
+ ,ss_ext_sales_price - COALESCE(sr_return_amt,0.0) AS sales_amt
+   FROM store_sales JOIN item ON i_item_sk=ss_item_sk
+JOIN date_dim ON d_date_sk=ss_sold_date_sk
+LEFT JOIN store_returns ON 
(ss_ticket_number=sr_ticket_

[05/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query86.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query86.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query86.q.out
new file mode 100644
index 000..c5b25e8
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query86.q.out
@@ -0,0 +1,240 @@
+PREHOOK: query: explain
+select   
+sum(ws_net_paid) as total_sum
+   ,i_category
+   ,i_class
+   ,grouping(i_category)+grouping(i_class) as lochierarchy
+   ,rank() over (
+   partition by grouping(i_category)+grouping(i_class),
+   case when grouping(i_class) = 0 then i_category end 
+   order by sum(ws_net_paid) desc) as rank_within_parent
+ from
+web_sales
+   ,date_dim   d1
+   ,item
+ where
+d1.d_month_seq between 1212 and 1212+11
+ and d1.d_date_sk = ws_sold_date_sk
+ and i_item_sk  = ws_item_sk
+ group by rollup(i_category,i_class)
+ order by
+   lochierarchy desc,
+   case when lochierarchy = 0 then i_category end,
+   rank_within_parent
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select   
+sum(ws_net_paid) as total_sum
+   ,i_category
+   ,i_class
+   ,grouping(i_category)+grouping(i_class) as lochierarchy
+   ,rank() over (
+   partition by grouping(i_category)+grouping(i_class),
+   case when grouping(i_class) = 0 then i_category end 
+   order by sum(ws_net_paid) desc) as rank_within_parent
+ from
+web_sales
+   ,date_dim   d1
+   ,item
+ where
+d1.d_month_seq between 1212 and 1212+11
+ and d1.d_date_sk = ws_sold_date_sk
+ and i_item_sk  = ws_item_sk
+ group by rollup(i_category,i_class)
+ order by
+   lochierarchy desc,
+   case when lochierarchy = 0 then i_category end,
+   rank_within_parent
+ limit 100
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+  Stage-2 is a root stage
+  Stage-1 depends on stages: Stage-2
+  Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+  Stage: Stage-2
+Spark
+ A masked pattern was here 
+  Vertices:
+Map 6 
+Map Operator Tree:
+TableScan
+  alias: d1
+  Statistics: Num rows: 73049 Data size: 81741831 Basic stats: 
COMPLETE Column stats: NONE
+  Filter Operator
+predicate: (d_date_sk is not null and d_month_seq BETWEEN 
1212 AND 1223) (type: boolean)
+Statistics: Num rows: 8116 Data size: 9081804 Basic stats: 
COMPLETE Column stats: NONE
+Select Operator
+  expressions: d_date_sk (type: int)
+  outputColumnNames: _col0
+  Statistics: Num rows: 8116 Data size: 9081804 Basic 
stats: COMPLETE Column stats: NONE
+  Spark HashTable Sink Operator
+keys:
+  0 _col0 (type: int)
+  1 _col0 (type: int)
+Local Work:
+  Map Reduce Local Work
+
+  Stage: Stage-1
+Spark
+  Edges:
+Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 174), Map 7 (PARTITION-LEVEL 
SORT, 174)
+Reducer 3 <- Reducer 2 (GROUP, 556)
+Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 278)
+Reducer 5 <- Reducer 4 (SORT, 1)
+ A masked pattern was here 
+  Vertices:
+Map 1 
+Map Operator Tree:
+TableScan
+  alias: web_sales
+  Statistics: Num rows: 144002668 Data size: 19580198212 Basic 
stats: COMPLETE Column stats: NONE
+  Filter Operator
+predicate: (ws_item_sk is not null and ws_sold_date_sk is 
not null) (type: boolean)
+Statistics: Num rows: 144002668 Data size: 19580198212 
Basic stats: COMPLETE Column stats: NONE
+Select Operator
+  expressions: ws_sold_date_sk (type: int), ws_item_sk 
(type: int), ws_net_paid (type: decimal(7,2))
+  outputColumnNames: _col0, _col1, _col2
+  Statistics: Num rows: 144002668 Data size: 19580198212 
Basic stats: COMPLETE Column stats: NONE
+  Map Join Operator
+condition map:
+ Inner Join 0 to 1
+keys:
+  0 _col0 (type: int)
+  1 _col0 (type: int)
+outputColumnNames: _col1, _col2
+input vertices:
+  1 Map 6
+Statistics: Num rows: 158402938 Data size: 21538218500 
Basic stats: COMPLETE Column stats: NONE
+Reduce Output Operator
+  key expressions: _col1 (type: int)
+  sort order: +
+  Map-reduce partition columns: _col1 (type: int)
+

[20/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query5.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query5.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query5.q.out
new file mode 100644
index 000..14e0bdb
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query5.q.out
@@ -0,0 +1,740 @@
+PREHOOK: query: explain
+with ssr as
+ (select s_store_id,
+sum(sales_price) as sales,
+sum(profit) as profit,
+sum(return_amt) as returns,
+sum(net_loss) as profit_loss
+ from
+  ( select  ss_store_sk as store_sk,
+ss_sold_date_sk  as date_sk,
+ss_ext_sales_price as sales_price,
+ss_net_profit as profit,
+cast(0 as decimal(7,2)) as return_amt,
+cast(0 as decimal(7,2)) as net_loss
+from store_sales
+union all
+select sr_store_sk as store_sk,
+   sr_returned_date_sk as date_sk,
+   cast(0 as decimal(7,2)) as sales_price,
+   cast(0 as decimal(7,2)) as profit,
+   sr_return_amt as return_amt,
+   sr_net_loss as net_loss
+from store_returns
+   ) salesreturns,
+ date_dim,
+ store
+ where date_sk = d_date_sk
+   and d_date between cast('1998-08-04' as date) 
+  and (cast('1998-08-04' as date) +  14 days)
+   and store_sk = s_store_sk
+ group by s_store_id)
+ ,
+ csr as
+ (select cp_catalog_page_id,
+sum(sales_price) as sales,
+sum(profit) as profit,
+sum(return_amt) as returns,
+sum(net_loss) as profit_loss
+ from
+  ( select  cs_catalog_page_sk as page_sk,
+cs_sold_date_sk  as date_sk,
+cs_ext_sales_price as sales_price,
+cs_net_profit as profit,
+cast(0 as decimal(7,2)) as return_amt,
+cast(0 as decimal(7,2)) as net_loss
+from catalog_sales
+union all
+select cr_catalog_page_sk as page_sk,
+   cr_returned_date_sk as date_sk,
+   cast(0 as decimal(7,2)) as sales_price,
+   cast(0 as decimal(7,2)) as profit,
+   cr_return_amount as return_amt,
+   cr_net_loss as net_loss
+from catalog_returns
+   ) salesreturns,
+ date_dim,
+ catalog_page
+ where date_sk = d_date_sk
+   and d_date between cast('1998-08-04' as date)
+  and (cast('1998-08-04' as date) +  14 days)
+   and page_sk = cp_catalog_page_sk
+ group by cp_catalog_page_id)
+ ,
+ wsr as
+ (select web_site_id,
+sum(sales_price) as sales,
+sum(profit) as profit,
+sum(return_amt) as returns,
+sum(net_loss) as profit_loss
+ from
+  ( select  ws_web_site_sk as wsr_web_site_sk,
+ws_sold_date_sk  as date_sk,
+ws_ext_sales_price as sales_price,
+ws_net_profit as profit,
+cast(0 as decimal(7,2)) as return_amt,
+cast(0 as decimal(7,2)) as net_loss
+from web_sales
+union all
+select ws_web_site_sk as wsr_web_site_sk,
+   wr_returned_date_sk as date_sk,
+   cast(0 as decimal(7,2)) as sales_price,
+   cast(0 as decimal(7,2)) as profit,
+   wr_return_amt as return_amt,
+   wr_net_loss as net_loss
+from web_returns left outer join web_sales on
+ ( wr_item_sk = ws_item_sk
+   and wr_order_number = ws_order_number)
+   ) salesreturns,
+ date_dim,
+ web_site
+ where date_sk = d_date_sk
+   and d_date between cast('1998-08-04' as date)
+  and (cast('1998-08-04' as date) +  14 days)
+   and wsr_web_site_sk = web_site_sk
+ group by web_site_id)
+  select  channel
+, id
+, sum(sales) as sales
+, sum(returns) as returns
+, sum(profit) as profit
+ from 
+ (select 'store channel' as channel
+, 'store' || s_store_id as id
+, sales
+, returns
+, (profit - profit_loss) as profit
+ from   ssr
+ union all
+ select 'catalog channel' as channel
+, 'catalog_page' || cp_catalog_page_id as id
+, sales
+, returns
+, (profit - profit_loss) as profit
+ from  csr
+ union all
+ select 'web channel' as channel
+, 'web_site' || web_site_id as id
+, sales
+, returns
+, (profit - profit_loss) as profit
+ from   wsr
+ ) x
+ group by rollup (channel, id)
+ order by channel
+ ,id
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with ssr as
+ (select s_store_id,
+sum(sales_price) as sales,
+sum(profit) as profit,
+sum(return_amt) as returns,
+sum(net_loss) as profit_loss
+ from
+  ( select  ss_store_sk as store_sk,
+ss_sold_date_sk  as date_sk,
+ss_ext_sales_price as sales_price,
+ss_net_profit as profit,
+cast(0 as decimal(7,2)) as return_amt,
+cast(0 as decimal(7,2)

[17/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query58.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query58.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query58.q.out
new file mode 100644
index 000..f06cbef
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query58.q.out
@@ -0,0 +1,944 @@
+Warning: Map Join MAPJOIN[183][bigTable=?] in task 'Stage-1:MAPRED' is a cross 
product
+Warning: Map Join MAPJOIN[184][bigTable=?] in task 'Stage-1:MAPRED' is a cross 
product
+Warning: Map Join MAPJOIN[185][bigTable=?] in task 'Stage-1:MAPRED' is a cross 
product
+PREHOOK: query: explain
+with ss_items as
+ (select i_item_id item_id
+,sum(ss_ext_sales_price) ss_item_rev 
+ from store_sales
+ ,item
+ ,date_dim
+ where ss_item_sk = i_item_sk
+   and d_date in (select d_date
+  from date_dim
+  where d_week_seq = (select d_week_seq 
+  from date_dim
+  where d_date = '1998-02-19'))
+   and ss_sold_date_sk   = d_date_sk
+ group by i_item_id),
+ cs_items as
+ (select i_item_id item_id
+,sum(cs_ext_sales_price) cs_item_rev
+  from catalog_sales
+  ,item
+  ,date_dim
+ where cs_item_sk = i_item_sk
+  and  d_date in (select d_date
+  from date_dim
+  where d_week_seq = (select d_week_seq 
+  from date_dim
+  where d_date = '1998-02-19'))
+  and  cs_sold_date_sk = d_date_sk
+ group by i_item_id),
+ ws_items as
+ (select i_item_id item_id
+,sum(ws_ext_sales_price) ws_item_rev
+  from web_sales
+  ,item
+  ,date_dim
+ where ws_item_sk = i_item_sk
+  and  d_date in (select d_date
+  from date_dim
+  where d_week_seq =(select d_week_seq 
+ from date_dim
+ where d_date = '1998-02-19'))
+  and ws_sold_date_sk   = d_date_sk
+ group by i_item_id)
+  select  ss_items.item_id
+   ,ss_item_rev
+   ,ss_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 ss_dev
+   ,cs_item_rev
+   ,cs_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 cs_dev
+   ,ws_item_rev
+   ,ws_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 ws_dev
+   ,(ss_item_rev+cs_item_rev+ws_item_rev)/3 average
+ from ss_items,cs_items,ws_items
+ where ss_items.item_id=cs_items.item_id
+   and ss_items.item_id=ws_items.item_id 
+   and ss_item_rev between 0.9 * cs_item_rev and 1.1 * cs_item_rev
+   and ss_item_rev between 0.9 * ws_item_rev and 1.1 * ws_item_rev
+   and cs_item_rev between 0.9 * ss_item_rev and 1.1 * ss_item_rev
+   and cs_item_rev between 0.9 * ws_item_rev and 1.1 * ws_item_rev
+   and ws_item_rev between 0.9 * ss_item_rev and 1.1 * ss_item_rev
+   and ws_item_rev between 0.9 * cs_item_rev and 1.1 * cs_item_rev
+ order by item_id
+ ,ss_item_rev
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with ss_items as
+ (select i_item_id item_id
+,sum(ss_ext_sales_price) ss_item_rev 
+ from store_sales
+ ,item
+ ,date_dim
+ where ss_item_sk = i_item_sk
+   and d_date in (select d_date
+  from date_dim
+  where d_week_seq = (select d_week_seq 
+  from date_dim
+  where d_date = '1998-02-19'))
+   and ss_sold_date_sk   = d_date_sk
+ group by i_item_id),
+ cs_items as
+ (select i_item_id item_id
+,sum(cs_ext_sales_price) cs_item_rev
+  from catalog_sales
+  ,item
+  ,date_dim
+ where cs_item_sk = i_item_sk
+  and  d_date in (select d_date
+  from date_dim
+  where d_week_seq = (select d_week_seq 
+  from date_dim
+  where d_date = '1998-02-19'))
+  and  cs_sold_date_sk = d_date_sk
+ group by i_item_id),
+ ws_items as
+ (select i_item_id item_id
+,sum(ws_ext_sales_price) ws_item_rev
+  from web_sales
+  ,item
+  ,date_dim
+ where ws_item_sk = i_item_sk
+  and  d_date in (select d_date
+  from date_dim
+  where d_week_seq =(select d_week_seq 
+ from date_dim
+ where d_date = '1998-02-19'))
+  and ws_sold_date_sk   = d_date_sk
+ group by i_item_id)
+  select  ss_items.item_id
+   ,ss_item_rev
+   ,ss_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 ss_dev
+   ,cs_item_rev
+   ,cs_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 cs_dev
+   ,ws_item_rev
+   ,ws_item_rev/(ss_item_rev+cs_item_rev+ws_item_rev)/3 * 100 ws_dev
+   ,(ss_item_rev+cs_item_rev+ws_item_rev)/3 average
+ from ss_items,cs_items,

[14/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query66.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query66.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query66.q.out
new file mode 100644
index 000..1dc0fac
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query66.q.out
@@ -0,0 +1,873 @@
+PREHOOK: query: explain
+select   
+ w_warehouse_name
+   ,w_warehouse_sq_ft
+   ,w_city
+   ,w_county
+   ,w_state
+   ,w_country
+,ship_carriers
+,year
+   ,sum(jan_sales) as jan_sales
+   ,sum(feb_sales) as feb_sales
+   ,sum(mar_sales) as mar_sales
+   ,sum(apr_sales) as apr_sales
+   ,sum(may_sales) as may_sales
+   ,sum(jun_sales) as jun_sales
+   ,sum(jul_sales) as jul_sales
+   ,sum(aug_sales) as aug_sales
+   ,sum(sep_sales) as sep_sales
+   ,sum(oct_sales) as oct_sales
+   ,sum(nov_sales) as nov_sales
+   ,sum(dec_sales) as dec_sales
+   ,sum(jan_sales/w_warehouse_sq_ft) as jan_sales_per_sq_foot
+   ,sum(feb_sales/w_warehouse_sq_ft) as feb_sales_per_sq_foot
+   ,sum(mar_sales/w_warehouse_sq_ft) as mar_sales_per_sq_foot
+   ,sum(apr_sales/w_warehouse_sq_ft) as apr_sales_per_sq_foot
+   ,sum(may_sales/w_warehouse_sq_ft) as may_sales_per_sq_foot
+   ,sum(jun_sales/w_warehouse_sq_ft) as jun_sales_per_sq_foot
+   ,sum(jul_sales/w_warehouse_sq_ft) as jul_sales_per_sq_foot
+   ,sum(aug_sales/w_warehouse_sq_ft) as aug_sales_per_sq_foot
+   ,sum(sep_sales/w_warehouse_sq_ft) as sep_sales_per_sq_foot
+   ,sum(oct_sales/w_warehouse_sq_ft) as oct_sales_per_sq_foot
+   ,sum(nov_sales/w_warehouse_sq_ft) as nov_sales_per_sq_foot
+   ,sum(dec_sales/w_warehouse_sq_ft) as dec_sales_per_sq_foot
+   ,sum(jan_net) as jan_net
+   ,sum(feb_net) as feb_net
+   ,sum(mar_net) as mar_net
+   ,sum(apr_net) as apr_net
+   ,sum(may_net) as may_net
+   ,sum(jun_net) as jun_net
+   ,sum(jul_net) as jul_net
+   ,sum(aug_net) as aug_net
+   ,sum(sep_net) as sep_net
+   ,sum(oct_net) as oct_net
+   ,sum(nov_net) as nov_net
+   ,sum(dec_net) as dec_net
+ from (
+(select 
+   w_warehouse_name
+   ,w_warehouse_sq_ft
+   ,w_city
+   ,w_county
+   ,w_state
+   ,w_country
+   ,'DIAMOND' || ',' || 'AIRBORNE' as ship_carriers
+   ,d_year as year
+   ,sum(case when d_moy = 1 
+   then ws_sales_price* ws_quantity else 0 end) as jan_sales
+   ,sum(case when d_moy = 2 
+   then ws_sales_price* ws_quantity else 0 end) as feb_sales
+   ,sum(case when d_moy = 3 
+   then ws_sales_price* ws_quantity else 0 end) as mar_sales
+   ,sum(case when d_moy = 4 
+   then ws_sales_price* ws_quantity else 0 end) as apr_sales
+   ,sum(case when d_moy = 5 
+   then ws_sales_price* ws_quantity else 0 end) as may_sales
+   ,sum(case when d_moy = 6 
+   then ws_sales_price* ws_quantity else 0 end) as jun_sales
+   ,sum(case when d_moy = 7 
+   then ws_sales_price* ws_quantity else 0 end) as jul_sales
+   ,sum(case when d_moy = 8 
+   then ws_sales_price* ws_quantity else 0 end) as aug_sales
+   ,sum(case when d_moy = 9 
+   then ws_sales_price* ws_quantity else 0 end) as sep_sales
+   ,sum(case when d_moy = 10 
+   then ws_sales_price* ws_quantity else 0 end) as oct_sales
+   ,sum(case when d_moy = 11
+   then ws_sales_price* ws_quantity else 0 end) as nov_sales
+   ,sum(case when d_moy = 12
+   then ws_sales_price* ws_quantity else 0 end) as dec_sales
+   ,sum(case when d_moy = 1 
+   then ws_net_paid_inc_tax * ws_quantity else 0 end) as jan_net
+   ,sum(case when d_moy = 2
+   then ws_net_paid_inc_tax * ws_quantity else 0 end) as feb_net
+   ,sum(case when d_moy = 3 
+   then ws_net_paid_inc_tax * ws_quantity else 0 end) as mar_net
+   ,sum(case when d_moy = 4 
+   then ws_net_paid_inc_tax * ws_quantity else 0 end) as apr_net
+   ,sum(case when d_moy = 5 
+   then ws_net_paid_inc_tax * ws_quantity else 0 end) as may_net
+   ,sum(case when d_moy = 6 
+   then ws_net_paid_inc_tax * ws_quantity else 0 end) as jun_net
+   ,sum(case when d_moy = 7 
+   then ws_net_paid_inc_tax * ws_quantity else 0 end) as jul_net
+   ,sum(case when d_moy = 8 
+   then ws_net_paid_inc_tax * ws_quantity else 0 end) as aug_net
+   ,sum(case when d_moy = 9 
+   then ws_net_paid_inc_tax * ws_quantity else 0 end) as sep_net
+   ,sum(case when d_moy = 10 
+   then ws_net_paid_inc_tax * ws_quantity else 0 end) as oct_net
+   ,sum(case when d_moy = 11
+

[22/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query47.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query47.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query47.q.out
new file mode 100644
index 000..def85ad
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query47.q.out
@@ -0,0 +1,771 @@
+PREHOOK: query: explain
+with v1 as(
+ select i_category, i_brand,
+s_store_name, s_company_name,
+d_year, d_moy,
+sum(ss_sales_price) sum_sales,
+avg(sum(ss_sales_price)) over
+  (partition by i_category, i_brand,
+ s_store_name, s_company_name, d_year)
+  avg_monthly_sales,
+rank() over
+  (partition by i_category, i_brand,
+ s_store_name, s_company_name
+   order by d_year, d_moy) rn
+ from item, store_sales, date_dim, store
+ where ss_item_sk = i_item_sk and
+   ss_sold_date_sk = d_date_sk and
+   ss_store_sk = s_store_sk and
+   (
+ d_year = 2000 or
+ ( d_year = 2000-1 and d_moy =12) or
+ ( d_year = 2000+1 and d_moy =1)
+   )
+ group by i_category, i_brand,
+  s_store_name, s_company_name,
+  d_year, d_moy),
+ v2 as(
+ select v1.i_category
+,v1.d_year, v1.d_moy
+,v1.avg_monthly_sales
+,v1.sum_sales, v1_lag.sum_sales psum, v1_lead.sum_sales nsum
+ from v1, v1 v1_lag, v1 v1_lead
+ where v1.i_category = v1_lag.i_category and
+   v1.i_category = v1_lead.i_category and
+   v1.i_brand = v1_lag.i_brand and
+   v1.i_brand = v1_lead.i_brand and
+   v1.s_store_name = v1_lag.s_store_name and
+   v1.s_store_name = v1_lead.s_store_name and
+   v1.s_company_name = v1_lag.s_company_name and
+   v1.s_company_name = v1_lead.s_company_name and
+   v1.rn = v1_lag.rn + 1 and
+   v1.rn = v1_lead.rn - 1)
+  select  *
+ from v2
+ where  d_year = 2000 and
+avg_monthly_sales > 0 and
+case when avg_monthly_sales > 0 then abs(sum_sales - 
avg_monthly_sales) / avg_monthly_sales else null end > 0.1
+ order by sum_sales - avg_monthly_sales, 3
+ limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+with v1 as(
+ select i_category, i_brand,
+s_store_name, s_company_name,
+d_year, d_moy,
+sum(ss_sales_price) sum_sales,
+avg(sum(ss_sales_price)) over
+  (partition by i_category, i_brand,
+ s_store_name, s_company_name, d_year)
+  avg_monthly_sales,
+rank() over
+  (partition by i_category, i_brand,
+ s_store_name, s_company_name
+   order by d_year, d_moy) rn
+ from item, store_sales, date_dim, store
+ where ss_item_sk = i_item_sk and
+   ss_sold_date_sk = d_date_sk and
+   ss_store_sk = s_store_sk and
+   (
+ d_year = 2000 or
+ ( d_year = 2000-1 and d_moy =12) or
+ ( d_year = 2000+1 and d_moy =1)
+   )
+ group by i_category, i_brand,
+  s_store_name, s_company_name,
+  d_year, d_moy),
+ v2 as(
+ select v1.i_category
+,v1.d_year, v1.d_moy
+,v1.avg_monthly_sales
+,v1.sum_sales, v1_lag.sum_sales psum, v1_lead.sum_sales nsum
+ from v1, v1 v1_lag, v1 v1_lead
+ where v1.i_category = v1_lag.i_category and
+   v1.i_category = v1_lead.i_category and
+   v1.i_brand = v1_lag.i_brand and
+   v1.i_brand = v1_lead.i_brand and
+   v1.s_store_name = v1_lag.s_store_name and
+   v1.s_store_name = v1_lead.s_store_name and
+   v1.s_company_name = v1_lag.s_company_name and
+   v1.s_company_name = v1_lead.s_company_name and
+   v1.rn = v1_lag.rn + 1 and
+   v1.rn = v1_lead.rn - 1)
+  select  *
+ from v2
+ where  d_year = 2000 and
+avg_monthly_sales > 0 and
+case when avg_monthly_sales > 0 then abs(sum_sales - 
avg_monthly_sales) / avg_monthly_sales else null end > 0.1
+ order by sum_sales - avg_monthly_sales, 3
+ limit 100
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+  Stage-2 is a root stage
+  Stage-3 depends on stages: Stage-2
+  Stage-4 depends on stages: Stage-3
+  Stage-1 depends on stages: Stage-4
+  Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+  Stage: Stage-2
+Spark
+ A masked pattern was here 
+  Vertices:
+Map 10 
+Map Operator Tree:
+TableScan
+  alias: store
+  Statistics: Num rows: 1704 Data size: 3256276 Basic stats: 
COMPLETE Column stats: NONE
+  Filter Operator
+predicate: (s_company_name is not null and s_store_name is 
not null and s_store_sk is not null) (type: boolean)
+Statistics: Num rows: 1704 Data size: 3256276 Basic stats: 
COMPLETE Column stats: NONE
+Select Operator
+  expressions: s_store_sk

[04/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query88.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query88.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query88.q.out
new file mode 100644
index 000..be44719
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query88.q.out
@@ -0,0 +1,1294 @@
+Warning: Map Join MAPJOIN[251][bigTable=?] in task 'Stage-1:MAPRED' is a cross 
product
+PREHOOK: query: explain
+select  *
+from
+ (select count(*) h8_30_to_9
+ from store_sales, household_demographics , time_dim, store
+ where ss_sold_time_sk = time_dim.t_time_sk   
+ and ss_hdemo_sk = household_demographics.hd_demo_sk 
+ and ss_store_sk = s_store_sk
+ and time_dim.t_hour = 8
+ and time_dim.t_minute >= 30
+ and ((household_demographics.hd_dep_count = 3 and 
household_demographics.hd_vehicle_count<=3+2) or
+  (household_demographics.hd_dep_count = 0 and 
household_demographics.hd_vehicle_count<=0+2) or
+  (household_demographics.hd_dep_count = 1 and 
household_demographics.hd_vehicle_count<=1+2)) 
+ and store.s_store_name = 'ese') s1,
+ (select count(*) h9_to_9_30 
+ from store_sales, household_demographics , time_dim, store
+ where ss_sold_time_sk = time_dim.t_time_sk
+ and ss_hdemo_sk = household_demographics.hd_demo_sk
+ and ss_store_sk = s_store_sk 
+ and time_dim.t_hour = 9 
+ and time_dim.t_minute < 30
+ and ((household_demographics.hd_dep_count = 3 and 
household_demographics.hd_vehicle_count<=3+2) or
+  (household_demographics.hd_dep_count = 0 and 
household_demographics.hd_vehicle_count<=0+2) or
+  (household_demographics.hd_dep_count = 1 and 
household_demographics.hd_vehicle_count<=1+2))
+ and store.s_store_name = 'ese') s2,
+ (select count(*) h9_30_to_10 
+ from store_sales, household_demographics , time_dim, store
+ where ss_sold_time_sk = time_dim.t_time_sk
+ and ss_hdemo_sk = household_demographics.hd_demo_sk
+ and ss_store_sk = s_store_sk
+ and time_dim.t_hour = 9
+ and time_dim.t_minute >= 30
+ and ((household_demographics.hd_dep_count = 3 and 
household_demographics.hd_vehicle_count<=3+2) or
+  (household_demographics.hd_dep_count = 0 and 
household_demographics.hd_vehicle_count<=0+2) or
+  (household_demographics.hd_dep_count = 1 and 
household_demographics.hd_vehicle_count<=1+2))
+ and store.s_store_name = 'ese') s3,
+ (select count(*) h10_to_10_30
+ from store_sales, household_demographics , time_dim, store
+ where ss_sold_time_sk = time_dim.t_time_sk
+ and ss_hdemo_sk = household_demographics.hd_demo_sk
+ and ss_store_sk = s_store_sk
+ and time_dim.t_hour = 10 
+ and time_dim.t_minute < 30
+ and ((household_demographics.hd_dep_count = 3 and 
household_demographics.hd_vehicle_count<=3+2) or
+  (household_demographics.hd_dep_count = 0 and 
household_demographics.hd_vehicle_count<=0+2) or
+  (household_demographics.hd_dep_count = 1 and 
household_demographics.hd_vehicle_count<=1+2))
+ and store.s_store_name = 'ese') s4,
+ (select count(*) h10_30_to_11
+ from store_sales, household_demographics , time_dim, store
+ where ss_sold_time_sk = time_dim.t_time_sk
+ and ss_hdemo_sk = household_demographics.hd_demo_sk
+ and ss_store_sk = s_store_sk
+ and time_dim.t_hour = 10 
+ and time_dim.t_minute >= 30
+ and ((household_demographics.hd_dep_count = 3 and 
household_demographics.hd_vehicle_count<=3+2) or
+  (household_demographics.hd_dep_count = 0 and 
household_demographics.hd_vehicle_count<=0+2) or
+  (household_demographics.hd_dep_count = 1 and 
household_demographics.hd_vehicle_count<=1+2))
+ and store.s_store_name = 'ese') s5,
+ (select count(*) h11_to_11_30
+ from store_sales, household_demographics , time_dim, store
+ where ss_sold_time_sk = time_dim.t_time_sk
+ and ss_hdemo_sk = household_demographics.hd_demo_sk
+ and ss_store_sk = s_store_sk 
+ and time_dim.t_hour = 11
+ and time_dim.t_minute < 30
+ and ((household_demographics.hd_dep_count = 3 and 
household_demographics.hd_vehicle_count<=3+2) or
+  (household_demographics.hd_dep_count = 0 and 
household_demographics.hd_vehicle_count<=0+2) or
+  (household_demographics.hd_dep_count = 1 and 
household_demographics.hd_vehicle_count<=1+2))
+ and store.s_store_name = 'ese') s6,
+ (select count(*) h11_30_to_12
+ from store_sales, household_demographics , time_dim, store
+ where ss_sold_time_sk = time_dim.t_time_sk
+ and ss_hdemo_sk = household_demographics.hd_demo_sk
+ and ss_store_sk = s_store_sk
+ and time_dim.t_hour = 11
+ and time_dim.t_minute >= 30
+ and ((household_demographics.hd_dep_count = 3 and 
household_demographics.hd_vehicle_count<=3+2) or
+  (household_demographics.hd_dep_count = 0 and 
household_demogr

[28/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query28.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query28.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query28.q.out
new file mode 100644
index 000..fb7e198
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query28.q.out
@@ -0,0 +1,517 @@
+Warning: Map Join MAPJOIN[94][bigTable=?] in task 'Stage-1:MAPRED' is a cross 
product
+PREHOOK: query: explain
+select  *
+from (select avg(ss_list_price) B1_LP
+,count(ss_list_price) B1_CNT
+,count(distinct ss_list_price) B1_CNTD
+  from store_sales
+  where ss_quantity between 0 and 5
+and (ss_list_price between 11 and 11+10 
+ or ss_coupon_amt between 460 and 460+1000
+ or ss_wholesale_cost between 14 and 14+20)) B1,
+ (select avg(ss_list_price) B2_LP
+,count(ss_list_price) B2_CNT
+,count(distinct ss_list_price) B2_CNTD
+  from store_sales
+  where ss_quantity between 6 and 10
+and (ss_list_price between 91 and 91+10
+  or ss_coupon_amt between 1430 and 1430+1000
+  or ss_wholesale_cost between 32 and 32+20)) B2,
+ (select avg(ss_list_price) B3_LP
+,count(ss_list_price) B3_CNT
+,count(distinct ss_list_price) B3_CNTD
+  from store_sales
+  where ss_quantity between 11 and 15
+and (ss_list_price between 66 and 66+10
+  or ss_coupon_amt between 920 and 920+1000
+  or ss_wholesale_cost between 4 and 4+20)) B3,
+ (select avg(ss_list_price) B4_LP
+,count(ss_list_price) B4_CNT
+,count(distinct ss_list_price) B4_CNTD
+  from store_sales
+  where ss_quantity between 16 and 20
+and (ss_list_price between 142 and 142+10
+  or ss_coupon_amt between 3054 and 3054+1000
+  or ss_wholesale_cost between 80 and 80+20)) B4,
+ (select avg(ss_list_price) B5_LP
+,count(ss_list_price) B5_CNT
+,count(distinct ss_list_price) B5_CNTD
+  from store_sales
+  where ss_quantity between 21 and 25
+and (ss_list_price between 135 and 135+10
+  or ss_coupon_amt between 14180 and 14180+1000
+  or ss_wholesale_cost between 38 and 38+20)) B5,
+ (select avg(ss_list_price) B6_LP
+,count(ss_list_price) B6_CNT
+,count(distinct ss_list_price) B6_CNTD
+  from store_sales
+  where ss_quantity between 26 and 30
+and (ss_list_price between 28 and 28+10
+  or ss_coupon_amt between 2513 and 2513+1000
+  or ss_wholesale_cost between 42 and 42+20)) B6
+limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select  *
+from (select avg(ss_list_price) B1_LP
+,count(ss_list_price) B1_CNT
+,count(distinct ss_list_price) B1_CNTD
+  from store_sales
+  where ss_quantity between 0 and 5
+and (ss_list_price between 11 and 11+10 
+ or ss_coupon_amt between 460 and 460+1000
+ or ss_wholesale_cost between 14 and 14+20)) B1,
+ (select avg(ss_list_price) B2_LP
+,count(ss_list_price) B2_CNT
+,count(distinct ss_list_price) B2_CNTD
+  from store_sales
+  where ss_quantity between 6 and 10
+and (ss_list_price between 91 and 91+10
+  or ss_coupon_amt between 1430 and 1430+1000
+  or ss_wholesale_cost between 32 and 32+20)) B2,
+ (select avg(ss_list_price) B3_LP
+,count(ss_list_price) B3_CNT
+,count(distinct ss_list_price) B3_CNTD
+  from store_sales
+  where ss_quantity between 11 and 15
+and (ss_list_price between 66 and 66+10
+  or ss_coupon_amt between 920 and 920+1000
+  or ss_wholesale_cost between 4 and 4+20)) B3,
+ (select avg(ss_list_price) B4_LP
+,count(ss_list_price) B4_CNT
+,count(distinct ss_list_price) B4_CNTD
+  from store_sales
+  where ss_quantity between 16 and 20
+and (ss_list_price between 142 and 142+10
+  or ss_coupon_amt between 3054 and 3054+1000
+  or ss_wholesale_cost between 80 and 80+20)) B4,
+ (select avg(ss_list_price) B5_LP
+,count(ss_list_price) B5_CNT
+,count(distinct ss_list_price) B5_CNTD
+  from store_sales
+  where ss_quantity between 21 and 25
+and (ss_list_price between 135 and 135+10
+  or ss_coupon_amt between 14180 and 14180+1000
+  or ss_wholesale_cost between 38 and 38+20)) B5,
+ (select avg(ss_list_price) B6_LP
+,count(ss_list_price) B6_CNT
+,count(distinct ss_list_price) B6_CNTD
+  from store_sales
+  where ss_quantity between 26 and 30
+and (ss_list_price between 28 and 28+10
+  or ss_coupon_amt between 2513 and 2513+1000
+  or ss_wholesale_cost be

[02/35] hive git commit: HIVE-17543: Enable PerfCliDriver for HoS (Sahil Takiar, reviewed by Peter Vary)

2017-09-28 Thread stakiar

http://git-wip-us.apache.org/repos/asf/hive/blob/bd371246/ql/src/test/results/clientpositive/perf/spark/query92.q.out
--
diff --git a/ql/src/test/results/clientpositive/perf/spark/query92.q.out 
b/ql/src/test/results/clientpositive/perf/spark/query92.q.out
new file mode 100644
index 000..677a85c
--- /dev/null
+++ b/ql/src/test/results/clientpositive/perf/spark/query92.q.out
@@ -0,0 +1,303 @@
+PREHOOK: query: explain
+select  
+   sum(ws_ext_discount_amt)  as `Excess Discount Amount` 
+from 
+web_sales 
+   ,item 
+   ,date_dim
+where
+i_manufact_id = 269
+and i_item_sk = ws_item_sk 
+and d_date between '1998-03-18' and 
+(cast('1998-03-18' as date) + 90 days)
+and d_date_sk = ws_sold_date_sk 
+and ws_ext_discount_amt  
+ > ( 
+ SELECT 
+1.3 * avg(ws_ext_discount_amt) 
+ FROM 
+web_sales 
+   ,date_dim
+ WHERE 
+  ws_item_sk = i_item_sk 
+  and d_date between '1998-03-18' and
+ (cast('1998-03-18' as date) + 90 days)
+  and d_date_sk = ws_sold_date_sk 
+  ) 
+order by sum(ws_ext_discount_amt)
+limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select  
+   sum(ws_ext_discount_amt)  as `Excess Discount Amount` 
+from 
+web_sales 
+   ,item 
+   ,date_dim
+where
+i_manufact_id = 269
+and i_item_sk = ws_item_sk 
+and d_date between '1998-03-18' and 
+(cast('1998-03-18' as date) + 90 days)
+and d_date_sk = ws_sold_date_sk 
+and ws_ext_discount_amt  
+ > ( 
+ SELECT 
+1.3 * avg(ws_ext_discount_amt) 
+ FROM 
+web_sales 
+   ,date_dim
+ WHERE 
+  ws_item_sk = i_item_sk 
+  and d_date between '1998-03-18' and
+ (cast('1998-03-18' as date) + 90 days)
+  and d_date_sk = ws_sold_date_sk 
+  ) 
+order by sum(ws_ext_discount_amt)
+limit 100
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+  Stage-2 is a root stage
+  Stage-3 depends on stages: Stage-2
+  Stage-1 depends on stages: Stage-3
+  Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+  Stage: Stage-2
+Spark
+ A masked pattern was here 
+  Vertices:
+Map 5 
+Map Operator Tree:
+TableScan
+  alias: date_dim
+  Statistics: Num rows: 73049 Data size: 81741831 Basic stats: 
COMPLETE Column stats: NONE
+  Filter Operator
+predicate: (CAST( d_date AS TIMESTAMP) BETWEEN 1998-03-18 
00:00:00.0 AND 1998-06-16 01:00:00.0 and d_date_sk is not null) (type: boolean)
+Statistics: Num rows: 8116 Data size: 9081804 Basic stats: 
COMPLETE Column stats: NONE
+Select Operator
+  expressions: d_date_sk (type: int)
+  outputColumnNames: _col0
+  Statistics: Num rows: 8116 Data size: 9081804 Basic 
stats: COMPLETE Column stats: NONE
+  Spark HashTable Sink Operator
+keys:
+  0 _col0 (type: int)
+  1 _col0 (type: int)
+Local Work:
+  Map Reduce Local Work
+
+  Stage: Stage-3
+Spark
+ A masked pattern was here 
+  Vertices:
+Map 9 
+Map Operator Tree:
+TableScan
+  alias: date_dim
+  Statistics: Num rows: 73049 Data size: 81741831 Basic stats: 
COMPLETE Column stats: NONE
+  Filter Operator
+predicate: (CAST( d_date AS TIMESTAMP) BETWEEN 1998-03-18 
00:00:00.0 AND 1998-06-16 01:00:00.0 and d_date_sk is not null) (type: boolean)
+Statistics: Num rows: 8116 Data size: 9081804 Basic stats: 
COMPLETE Column stats: NONE
+Select Operator
+  expressions: d_date_sk (type: int)
+  outputColumnNames: _col0
+  Statistics: Num rows: 8116 Data size: 9081804 Basic 
stats: COMPLETE Column stats: NONE
+  Spark HashTable Sink Operator
+keys:
+  0 _col0 (type: int)
+  1 _col0 (type: int)
+Local Work:
+  Map Reduce Local Work
+
+  Stage: Stage-1
+Spark
+  Edges:
+Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 261), Reducer 8 
(PARTITION-LEVEL SORT, 261)
+Reducer 3 <- Reducer 2 (GROUP, 1)
+Reducer 4 <- Reducer 3 (SORT, 1)
+Reducer 7 <- Map 6 (GROUP, 169)
+Reducer 8 <- Map 10 (PARTITION-LEVEL SORT, 87), Reducer 7 
(PARTITION-LEVEL SORT, 87)
+ A masked pattern was here 
+  Vertices:
+Map 1 
+Map Operator Tree:
+TableScan
+  alias: web_sales
+  Statistics: Num rows: 144002668 Data size: 195801982

< 1 2 3 >

101 - 200 of 242 matches

Mail list logo