[jira] [Created] (HIVE-15114) Remove extra MoveTask operators

2016-11-02 Thread Sahil Takiar (JIRA) 
Sahil Takiar created HIVE-15114:
---

 Summary: Remove extra MoveTask operators
 Key: HIVE-15114
 URL: https://issues.apache.org/jira/browse/HIVE-15114
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 2.1.0
Reporter: Sahil Takiar


When running simple insert queries (e.g. {{INSERT INTO TABLE ... VALUES ...}}) 
there an extraneous {{MoveTask}s is created.

This is problematic when the scratch directory is on S3 since renames require 
copying the entire dataset.

For simple queries (like the one above), there are two MoveTasks. The first one 
moves the output data from one file in the scratch directory to another file in 
the scratch directory. The second MoveTask moves the data from the scratch 
directory to its final table location.

The first MoveTask should not be necessary. The goal of this JIRA it to remove 
it. This should help improve performance when running on S3.

It seems that the first Move might be caused by a dependency resolution problem 
in the optimizer, where a dependent task doesn't get properly removed when the 
task it depends on is filtered by a condition resolver.

A dummy {{MoveTask}} is added in the 
{{GenMapRedUtils.createMRWorkForMergingFiles}} method. This method creates a 
conditional task which launches a job to merge tasks at the end of the file. At 
the end of the conditional job there is a MoveTask.

Even though Hive decides that the conditional merge job is no needed, it seems 
the MoveTask is still added to the plan.

Seems this extra {{MoveTask}} may have been added intentionally. Not sure why 
yet. The {{ConditionalResolverMergeFiles}} says that one of three tasks will be 
returned: move task only, merge task only, merge task followed by a move task.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15113) SHOW CREATE TABLE on skewed table returns statement without skew definition

2016-11-02 Thread Wojciech Meler (JIRA) 
Wojciech Meler created HIVE-15113:
-

 Summary: SHOW CREATE TABLE on skewed table returns statement 
without skew definition
 Key: HIVE-15113
 URL: https://issues.apache.org/jira/browse/HIVE-15113
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Wojciech Meler


CREATE TABLE IF NOT EXISTS testskew (key int, value STRING)
SKEWED BY (key) ON (1,5,6) STORED AS DIRECTORIES
STORED AS ORC;

SHOW CREATE TABLE testskew;

CREATE TABLE `testskew`(
2 `key` int, 
3 `value` string)
4   ROW FORMAT SERDE 
5 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' 
6   STORED AS INPUTFORMAT 
7 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' 
8   OUTPUTFORMAT 
9 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
10  LOCATION
11'hdfs://nameservice1/user/hive/warehouse/private_wmeler.db/testskew'
12  TBLPROPERTIES (
13'COLUMN_STATS_ACCURATE'='true', 
14'numFiles'='4', 
15'numRows'='19', 
16'rawDataSize'='1736', 
17'totalSize'='1184', 
18'transient_lastDdlTime'='1478098814')



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15112) Implement Parquet vectorization reader for Complex types

2016-11-02 Thread Ferdinand Xu (JIRA) 
Ferdinand Xu created HIVE-15112:
---

 Summary: Implement Parquet vectorization reader for Complex types 
 Key: HIVE-15112
 URL: https://issues.apache.org/jira/browse/HIVE-15112
 Project: Hive
  Issue Type: Sub-task
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu


Like HIVE-14815, we need support Parquet vectorized reader for complex types 
like map, struct and union as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 53166: HIVE-14910: Flaky test: TestSparkClient.testJobSubmission

2016-11-02 Thread Barna Zsombor Klara 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53166/
---

(Updated Nov. 2, 2016, 9:35 a.m.)


Review request for hive, Mohit Sabharwal, Siddharth Seth, and Xuefu Zhang.


Changes
---

Renamed the variables in JobHandleImp as requested in the review.


Repository: hive-git


Description
---

HIVE-14910: Flaky test: TestSparkClient.testJobSubmission
I ran into this problem today while investigating a flaky test. I think the 
failure is coming from this race condition: the listener can be added to the 
JobHandle only after the job has been submitted. So there is no guarantee that 
every method of the listener will be invoked, some state changes may have 
happened before the caller received the handler back.
I propose a slight change in the API. We should add the listeners as an 
argument of the submit method, so we can set them on the Handler before the job 
itself is submitted. This way any status change should be signalled to the 
listener.


Diffs (updated)
-

  spark-client/src/main/java/org/apache/hive/spark/client/JobHandle.java 
44aa255a8271894ed3e787c3e7d1323628db63c4 
  spark-client/src/main/java/org/apache/hive/spark/client/JobHandleImpl.java 
17c8f40edd472682d5604f41980d06e60cc92893 
  spark-client/src/main/java/org/apache/hive/spark/client/SparkClient.java 
3e921a5d9b77966d368684ee7b6f1c861ac60e08 
  spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java 
e2a30a76e0f7fe95d8a453f502311baa08abcbe2 
  spark-client/src/test/java/org/apache/hive/spark/client/TestJobHandle.java 
e8f352dce9f618573c2d79e9b8c59e19fad7298a 
  spark-client/src/test/java/org/apache/hive/spark/client/TestSparkClient.java 
b95cd7a05d44224b53bf2cef9170146b8b2eb4a8 

Diff: https://reviews.apache.org/r/53166/diff/


Testing
---

Unit tests modified and tested.
I also ran a simple query with HoS as the execution engine.


Thanks,

Barna Zsombor Klara

Snapshot builds are not deployed

2016-11-02 Thread Gary Gregory 
Hi All,

It looks like snapshot builds are not deployed to the snapshot repo.

For example, in
https://repository.apache.org/content/groups/snapshots/org/apache/hive/hive-jdbc/2.2.0-SNAPSHOT/
the last update was on October 13.

I am looking for a fix for https://issues.apache.org/jira/browse/HIVE-14837
which was committed on Oct 24 it seems. I cannot build locally either:
https://issues.apache.org/jira/browse/HIVE-15111

Thank you,
Gary


-- 
E-Mail: garydgreg...@gmail.com | ggreg...@apache.org  
Java Persistence with Hibernate, Second Edition



JUnit in Action, Second Edition



Spring Batch in Action



Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory

[jira] [Created] (HIVE-15111) Can't build on Windows from git master (2.2.0-SNAPSHOT)

2016-11-02 Thread Gary Gregory (JIRA) 
Gary Gregory created HIVE-15111:
---

 Summary: Can't build on Windows from git master (2.2.0-SNAPSHOT)
 Key: HIVE-15111
 URL: https://issues.apache.org/jira/browse/HIVE-15111
 Project: Hive
  Issue Type: Bug
 Environment: Apache Maven 3.3.9 
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T08:41:47-08:00)
Maven home: E:\Java\apache-maven-3.3.9\bin\..
Java version: 1.8.0_101, vendor: Oracle Corporation
Java home: C:\Program Files\Java\jdk1.8.0_101\jre
Default locale: en_US, platform encoding: Cp1252
OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 
2015-11-10T08:41:47-08:00)
Maven home: E:\Java\apache-maven-3.3.9\bin\..
Java version: 1.7.0_79, vendor: Oracle Corporation
Java home: C:\Program Files\Java\jdk1.7.0_79\jre
Default locale: en_US, platform encoding: Cp1252
OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"
Reporter: Gary Gregory


{noformat}
Tests in error:
  TestColumnStatistics.testHasNull:278 » NullPointer
  TestNewIntegerEncoding.testBasicDelta1:230 » NullPointer
  TestNewIntegerEncoding.testBasicDelta1:229 » FileAlreadyExists File already 
ex...
  TestNewIntegerEncoding.testBasicDelta2:264 » NullPointer
  TestNewIntegerEncoding.testBasicDelta2:263 » FileAlreadyExists File already 
ex...
  TestNewIntegerEncoding.testBasicDelta3:298 » NullPointer
  TestNewIntegerEncoding.testBasicDelta3:297 » FileAlreadyExists File already 
ex...
  TestNewIntegerEncoding.testBasicDelta4:332 » NullPointer
  TestNewIntegerEncoding.testBasicDelta4:331 » FileAlreadyExists File already 
ex...
  TestNewIntegerEncoding.testBasicNew:195 » FileAlreadyExists File already 
exist...
  TestNewIntegerEncoding.testBasicNew:195 » FileAlreadyExists File already 
exist...
  TestNewIntegerEncoding.testBasicOld:157 » FileAlreadyExists File already 
exist...
  TestNewIntegerEncoding.testBasicOld:157 » FileAlreadyExists File already 
exist...
  TestNewIntegerEncoding.testBasicRow:122 » FileAlreadyExists File already 
exist...
  TestNewIntegerEncoding.testBasicRow:122 » FileAlreadyExists File already 
exist...
  TestNewIntegerEncoding.testDeltaOverflow2:398 » NullPointer
  TestNewIntegerEncoding.testDeltaOverflow2:397 » FileAlreadyExists File 
already...
  TestNewIntegerEncoding.testDeltaOverflow3:430 » NullPointer
  TestNewIntegerEncoding.testDeltaOverflow3:429 » FileAlreadyExists File 
already...
  TestNewIntegerEncoding.testDeltaOverflow:364 » FileAlreadyExists File already 
...
  TestNewIntegerEncoding.testDeltaOverflow:364 » FileAlreadyExists File already 
...
  TestNewIntegerEncoding.testDirectLargeNegatives:1314 » FileAlreadyExists File 
...
  TestNewIntegerEncoding.testDirectLargeNegatives:1314 » FileAlreadyExists File 
...
  TestNewIntegerEncoding.testIntegerMax:496 » FileAlreadyExists File already 
exi...
  TestNewIntegerEncoding.testIntegerMax:496 » FileAlreadyExists File already 
exi...
  TestNewIntegerEncoding.testIntegerMin:462 » FileAlreadyExists File already 
exi...
  TestNewIntegerEncoding.testIntegerMin:462 » FileAlreadyExists File already 
exi...
  TestNewIntegerEncoding.testLongMax:564 » FileAlreadyExists File already 
exists...
  TestNewIntegerEncoding.testLongMax:564 » FileAlreadyExists File already 
exists...
  TestNewIntegerEncoding.testLongMin:530 » FileAlreadyExists File already 
exists...
  TestNewIntegerEncoding.testLongMin:530 » FileAlreadyExists File already 
exists...
  TestNewIntegerEncoding.testPatchedBase510:996 » NullPointer
  TestNewIntegerEncoding.testPatchedBase510:995 » FileAlreadyExists File 
already...
  TestNewIntegerEncoding.testPatchedBase511:1033 » NullPointer
  TestNewIntegerEncoding.testPatchedBase511:1032 » FileAlreadyExists File 
alread...
  TestNewIntegerEncoding.testPatchedBaseAt0:848 » NullPointer
  TestNewIntegerEncoding.testPatchedBaseAt0:847 » FileAlreadyExists File 
already...
  TestNewIntegerEncoding.testPatchedBaseAt1:886 » NullPointer
  TestNewIntegerEncoding.testPatchedBaseAt1:885 » FileAlreadyExists File 
already...
  TestNewIntegerEncoding.testPatchedBaseAt255:921 » FileAlreadyExists File 
alrea...
  TestNewIntegerEncoding.testPatchedBaseAt255:921 » FileAlreadyExists File 
alrea...
  TestNewIntegerEncoding.testPatchedBaseAt256:958 » FileAlreadyExists File 
alrea...
  TestNewIntegerEncoding.testPatchedBaseAt256:958 » FileAlreadyExists File 
alrea...
  TestNewIntegerEncoding.testPatchedBaseMax1:1069 » FileAlreadyExists File 
alrea...
  TestNewIntegerEncoding.testPatchedBaseMax1:1069 » FileAlreadyExists File 
alrea...
  TestNewIntegerEncoding.testPatchedBaseMax2:1108 » FileAlreadyExists File 
alrea...
  TestNewIntegerEncoding.testPatchedBaseMax2:1108 » FileAlreadyExists File 
alrea...
  TestNewIntegerEncoding.testPatchedBaseMax3:1160 » FileAlreadyExists File 
alrea...
  TestNewIntegerEncoding.testPatchedBaseMax3:1160 » NullPointer