[jira] [Created] (HIVE-12284) CLONE - Merge master to Spark branch 10/26/2015 [Spark Branch]

[Xuefu Zhang (JIRA)]

Xuefu Zhang created HIVE-12284:
--

 Summary: CLONE - Merge master to Spark branch 10/26/2015 [Spark 
Branch]
 Key: HIVE-12284
 URL: https://issues.apache.org/jira/browse/HIVE-12284
 Project: Hive
  Issue Type: Sub-task
  Components: Spark
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
 Fix For: spark-branch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12285) Add locking to HCatClient

[Elliot West (JIRA)]

Elliot West created HIVE-12285:
--

 Summary: Add locking to HCatClient
 Key: HIVE-12285
 URL: https://issues.apache.org/jira/browse/HIVE-12285
 Project: Hive
  Issue Type: Improvement
  Components: HCatalog
Affects Versions: 2.0.0
Reporter: Elliot West
Assignee: Elliot West


With the introduction of a concurrency model (HIVE-1293) Hive uses locks to 
coordinate  access and updates to both table data and metadata. Within the Hive 
CLI such lock management is seamless. However, Hive provides additional APIs 
that permit interaction with data repositories, namely the HCatalog APIs. 
Currently, operations implemented by this API do not participate with Hive's 
locking scheme. Furthermore, access to the locking mechanisms is not exposed by 
the APIs (as is the case with the Metastore Thrift API) and so users are not 
able to explicitly interact with locks either. This has created a less than 
ideal situation where users of the APIs have no choice but to manipulate these 
data repositories outside of the command of Hive's lock management, potentially 
resulting in situations where data inconsistencies can occur both for external 
processes using the API and for queries executing within Hive.

h3. Scope of work
This ticket is concerned with sections of the HCatalog API that deal with DDL 
type operations using the metastore, not with those whose purpose is to 
read/write table data. A separate issue already exists for adding locking to 
HCat readers and writers (HIVE-6207).

h3. Proposed work
The following work items would serve as a minimum deliverable that would both 
allow API users to effectively work with locks:
* Comprehensively document on the wiki the locks required for various Hive 
operations. At a minimum this should cover all operations exposed by 
{{HCatClient}}. The [Locking design 
document|https://cwiki.apache.org/confluence/display/Hive/Locking] can be used 
as a starting point or perhaps updated.
* Implement methods and types in the {{HCatClient}} API that allow users to 
manipulate Hive locks. For the most part I'd expect these to delegate to the 
metastore API implementations:
** {{org.apache.hadoop.hive.metastore.IMetaStoreClient.lock(LockRequest)}}
** {{org.apache.hadoop.hive.metastore.IMetaStoreClient.checkLock(long)}}
** {{org.apache.hadoop.hive.metastore.IMetaStoreClient.unlock(long)}}
** {{org.apache.hadoop.hive.metastore.IMetaStoreClient.showLocks()}}
** {{org.apache.hadoop.hive.metastore.IMetaStoreClient.heartbeat(long, long)}}
** {{org.apache.hadoop.hive.metastore.api.LockComponent}}
** {{org.apache.hadoop.hive.metastore.api.LockRequest}}
** {{org.apache.hadoop.hive.metastore.api.LockResponse}}
** {{org.apache.hadoop.hive.metastore.api.LockLevel}}
** {{org.apache.hadoop.hive.metastore.api.LockType}}
** {{org.apache.hadoop.hive.metastore.api.LockState}}
** {{org.apache.hadoop.hive.metastore.api.ShowLocksResponse}}

h3. Additional proposals
Explicit lock management should be fairly simple to add to {{HCatClient}}, 
however it puts the onus on the API user to correctly understand and implement 
code that uses lock in an appropriate manner. Failure to do so may have 
undesirable consequences. With a simpler user model the operations exposed on 
the API would automatically acquire and release the locks that they need. This 
might work well for small numbers of operations, but not perhaps for large 
sequences of invocations. (Do we need to worry about this though as the API 
methods usually accept batches?).  Additionally tasks such as heartbeat 
management could also be handled implicitly for long running sets of 
operations. With these concerns in mind it may also be beneficial to deliver 
some of the following:
* A means to automatically acquire/release appropriate locks for {{HCatClient}} 
operations.
* A component that maintains a lock heartbeat from the client.
* A strategy for switching between manual/automatic lock management, analogous 
to SQL's {{autocommit}} for transactions.

An API for lock and heartbeat management already exists in the HCatalog 
Mutation API (see: {{org.apache.hive.hcatalog.streaming.mutate.client.lock}}). 
It will likely make sense to refactor either this code and/or code that uses it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12286) Add option to ORC vectorized reader to not trim spaces from char columns.

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-12286:


 Summary: Add option to ORC vectorized reader to not trim spaces 
from char columns.
 Key: HIVE-12286
 URL: https://issues.apache.org/jira/browse/HIVE-12286
 Project: Hive
  Issue Type: Sub-task
Reporter: Owen O'Malley


Currently the ORC reader in nextBatch always strips spaces from char columns. 
It is more natural for non-Hive applications to make it not trim the results on 
read, so I propose adding a switch to ReaderOptions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12287) Lineage for lateral view shows wrong dependencies

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12287:
--

 Summary: Lineage for lateral view shows wrong dependencies
 Key: HIVE-12287
 URL: https://issues.apache.org/jira/browse/HIVE-12287
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


The lineage dependency graph for select from lateral view is wrong.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 39735: HIVE-12215 Exchange partition does not show outputs field for post/pre execute hooks

[Aihua Xu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39735/
---

Review request for hive.


Repository: hive-git


Description
---

HIVE-12215 Exchange partition does not show outputs field for post/pre execute 
hooks


Diffs
-

  metastore/if/hive_metastore.thrift 3e30f560d0ca604e3faca1e49c1dd97fcbe13489 
  metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 
c8f16a7fad67b4af899ee3ae84120f42fb7a7979 
  metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 
a82c363937f956371a5873bb4717a0f1f2153b75 
  metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 
9eca65c1c0ba64b6ac34738613d80f0cb5e98a7e 
  
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 0c67416473752e61565131f1e3a5ef6e107d3fb7 
  metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php 
e922d7d996fd995d7312809ebb6be054b603b8fa 
  metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 
8dba17b536f68605a84f4718920d6b614c35cf4d 
  metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 
59c7b94cb1bc8984d790eb6afa4577f11c0c9b4d 
  metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 
7b93158a2af3e58915eece19ded01f7a2019ec87 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
cf2e25be549ded128151343a8d5371f091bbf7d3 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
3960f5d99277186192f779dbf6f64bf3dfac8ad3 
  metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
f3a23f52e5f1893234b0890bc340663025dea2a5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
dcac9ca900e58ae6996657247e7d7931d5f67ac1 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
cef297a3f4368c9b2310d0949f940128db6f6b00 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
b4546e104f59879cbdc4d7e94dc17022ef700f53 
  ql/src/test/results/clientnegative/exchange_partition.q.out 
8622615ac3cbcdf8e2767b150efe023237162030 
  ql/src/test/results/clientpositive/exchange_partition.q.out 
5b21eaf72d52497e73ee7531c32636752751051f 
  ql/src/test/results/clientpositive/exchange_partition2.q.out 
8c7c583daa5852977b0b24178350d224e57a067e 
  ql/src/test/results/clientpositive/exchange_partition3.q.out 
3815861dbae5606415ad6565522dd07e6f3f9bf4 

Diff: https://reviews.apache.org/r/39735/diff/


Testing
---


Thanks,

Aihua Xu

Re: Review Request 37778: HIVE-11634

[Hari Sankar Sivarama Subramaniyan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/
---

(Updated Oct. 28, 2015, 10:48 p.m.)


Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
Pullokkaran.


Changes
---

Addressed John's comments from the previous upload.


Repository: hive-git


Description
---

Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 59b66cd 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 7ee5081 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
4799b4d 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
7cdc730 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
7262164 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
  ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
  ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
  ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
  ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
  ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out 
4320f01 
  
ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
 e30465d 
  ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
  ql/src/test/results/clientpositive/tez/dynamic_partition_pruning.q.out 
7e28c77 
  
ql/src/test/results/clientpositive/tez/vectorized_dynamic_partition_pruning.q.out
 1103e80 

Diff: https://reviews.apache.org/r/37778/diff/


Testing
---

Local testing done. More unit tests coming in the next patch.


Thanks,

Hari Sankar Sivarama Subramaniyan

Review Request 39741: HIVE-12280 : HiveConnection does not try all possible HS2 in dynamic service discovery mode.

[Szehon Ho]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39741/
---

Review request for hive.


Bugs: HIVE-12280
https://issues.apache.org/jira/browse/HIVE-12280


Repository: hive-git


Description
---

The client calls ZooKeeperHiveClientHelper.configureConnParams() on retry to 
get a new HS2 host.  But there are some checks to prevent the host/port from 
getting reset if they are already set, which will make the retries all try on 
the same host/port.


Diffs
-

  itests/hive-unit/pom.xml 326d646 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestServiceDiscovery.java 
PRE-CREATION 
  jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientHelper.java 6c21423 

Diff: https://reviews.apache.org/r/39741/diff/


Testing
---

Added unit test.


Thanks,

Szehon Ho

[jira] [Created] (HIVE-12288) Extend HIVE-11306 changes to apply to Native vectorized map-joins

[Gopal V (JIRA)]

Gopal V created HIVE-12288:
--

 Summary: Extend HIVE-11306 changes to apply to Native vectorized 
map-joins
 Key: HIVE-12288
 URL: https://issues.apache.org/jira/browse/HIVE-12288
 Project: Hive
  Issue Type: Bug
  Components: Vectorization
Affects Versions: 2.0.0
Reporter: Gopal V
Assignee: Gopal V
 Fix For: 2.0.0
 Attachments: HIVE-12288.1.patch

HIVE-11306 applies to the old style VectorMapJoin operators, while the 
specialized operators go via a different codepath into the 
HybridHybridHashTable.

Apply similar changes to the setDirect() codepath.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12289) Make sure slf4j-log4j12 jar is not in classpath

[Ashutosh Chauhan (JIRA)]

Ashutosh Chauhan created HIVE-12289:
---

 Summary: Make sure slf4j-log4j12 jar is not in classpath
 Key: HIVE-12289
 URL: https://issues.apache.org/jira/browse/HIVE-12289
 Project: Hive
  Issue Type: Improvement
  Components: Logging
Affects Versions: 2.0.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


log4j12 which is version 1.2 gets pulled in by transitive dependency. Need to 
make sure we only have log4j2 is in classpath, otherwise slf4j may bind to 
version 1.2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: HIVE-TRUNK-JAVA8 #127

[hiveqa]

See 


Changes:

[daijy] HIVE-12262: Session log dir cannot be created in some cases (Daniel 
Dai, reviewed by Thejas Nair)

[Ferdinand Xu] HIVE-12234: Beeline quit tries to connect again if no existing 
connections(Ferdinand Xu, reviewed by Szehon Ho)

[Chao Sun] HIVE-12189: The list in pushdownPreds of ppd.ExprWalkerInfo should 
not be allowed to grow very large (Yongzhi Chen, reviewed by Chao Sun)

[thejas] HIVE-12261 : schematool version info exit status should depend on 
compatibility, not equality (Thejas Nair, reviewed by Sushanth Sowmyan)

[thejas] HIVE-12261 - adding more comments

[j.prasanth.j] HIVE-12246: Orc FileDump fails with Missing CLI jar (Prasanth 
Jayachandran reviewed by Ferdinand Xu)

[j.prasanth.j] HIVE-12260: Fix TestColumnStatistics and TestJsonFileDump test 
failures in master (Prasanth Jayachandran reviewed by Thejas Nair)

[Aihua Xu] HIVE-12213 Investigating the test failure 
TestHCatClient.testTableSchemaPropagation (Aleksei Statkevich via Aihua Xu)

[khorgath] HIVE-9013 : Hive set command exposes metastore db password (Binglin 
Chang, reviewed by Thejas Nair, Sushanth Sowmyan)

[j.prasanth.j] HIVE-11523: org.apache.hadoop.hive.ql.io.orc.FileDump should 
handle errors (Prasanth Jayachandran reviewed by Ashutosh Chauhan)

[Alan Gates] HIVE-11378 Remove hadoop-1 support from master branch (gates, 
reviewed by Ashutosh Chauhan and Sergey Shelukhin)

[Ashutosh Chauhan] HIVE-10807 : Invalidate basic stats for insert queries if 
autogather=false (Ashutosh Chauhan via Gopal V)

[ctang] HIVE-12259: Command containing semicolon is broken in Beeline (Chaoyu 
Tang, reviewed by Ferdinand Xu)

[Jimmy Xiang] HIVE-11973: IN operator fails when the column type is DATE 
(Yongzhi via Jimmy)

[Szehon Ho] HIVE-12250 Zookeeper connection leaks in Hive's HBaseHandler 
(Naveen, reviewed by Aihua and Szehon)

[Szehon Ho] HIVE-11489 : Jenkins PreCommit-HIVE-SPARK-Build fails with 
TestCliDriver.initializationError (Szehon, reviewed by Sergio Pena)

[sershe] HIVE-12061 : add file type support to file metadata by expr call 
(Sergey Shelukhin, reviewed by Alan Gates)

[sershe] HIVE-12227 : LLAP: better column vector object pools (Sergey 
Shelukhin, reviewed by Gopal V)

[j.prasanth.j] HIVE-11497: Make sure --orcfiledump utility includes 
OrcRecordUpdate.AcidStats (Prasanth Jayachandran reviewed by Eugene Koifman)

--
[...truncated 310 lines...]
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testParallelFailsOnExec.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestPhase.testExecHostsWithFailure.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestCleanupPhase.testExecute.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testIsolatedFailsOnRsyncUnknown.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepNone.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testBasic.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/TestLocalCommand.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestExecutionPhase.testFailingQFile.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/MockLocalCommandFactory.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testAlternativeTestJVM.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/context
A
ptest2/src/test/java/org/apache/hive/ptest/execution/context/TestCloudComputeService.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/context/TestCloudExecutionContextProvider.java
Aptest2/src/test/java/org/apache/hive/ptest/execution/TestJIRAService.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestExecutionPhase.testFailingUnitTest.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testIsolatedFailsOnRsyncOne.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestHostExecutor.testShutdownBeforeExec.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestPhase.testExecInstancesWithFailure.approved.txt
A
ptest2/src/test/java/org/apache/hive/ptest/execution/TestScripts.testPrepGit.approved.txt
Aptest2/src/test/java/org/apache/hive/ptest/execution/ssh
A
ptest2/src/test/java/org/apache/hive/ptest/execution/ssh/TestRSyncCommandExecutor.java
A
ptest2/src/test/java/org/apache/hive/ptest/execution/ssh/TestSSHCommandExecutor.java
Aptest2/src/test/resources
Aptest2/src/test/resources/test-configuration.properties
Aptest2/src/test/resources/test-outputs
A
ptest2/src/test/resources/test-outputs/index_auth.q-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
A