date:20180525

Re: Review Request 67125: HIVE-19418 add background stats updater similar to compactor

2018-05-25 Thread Sergey Shelukhin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67125/
---

(Updated May 26, 2018, 1:30 a.m.)


Review request for hive, Ashutosh Chauhan and Seong (Steve) Yeom.


Repository: hive-git


Description
---

see jira. This should eventually integrate with ACID stats to determine what 
stats are out of date, when that is done. Probably in separate jira if this 
goes in first.


Diffs (updated)
-

  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 3d6fda6bd4 
  ql/src/java/org/apache/hadoop/hive/ql/DriverUtils.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 6c56212c9e 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
982b180761 
  ql/src/test/org/apache/hadoop/hive/ql/stats/TestStatsUpdaterThread.java 
PRE-CREATION 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
 9ab9e85742 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 92d2e3f368 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 5bb1985025 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 13ccdb145e 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 ce7d2861dd 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 b223920e82 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/EnumValidator.java
 PRE-CREATION 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 114d5da205 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 f6899be750 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 98a85cc758 


Diff: https://reviews.apache.org/r/67125/diff/2/

Changes: https://reviews.apache.org/r/67125/diff/1-2/


Testing
---


Thanks,

Sergey Shelukhin

Re: Review Request 67125: HIVE-19418 add background stats updater similar to compactor

2018-05-25 Thread Sergey Shelukhin



> On May 23, 2018, 9:43 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java
> > Lines 557 (patched)
> > 
> >
> > This will compute basic and column stats. I assume this is what you 
> > want.

Yes.


> On May 23, 2018, 9:43 p.m., Ashutosh Chauhan wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Lines 2424 (patched)
> > 
> >
> > Returning concatenated table Name + Dbname is error prone. Lets make 
> > this return List 

This seems to be a standard approach in Hive to pass around the full name(s). 
In fact some existing APIs for metastore return catalog+db in one string, and 
Hive uses full table name all over the place.
I don't think either is error-prone and requires String[]...


> On May 23, 2018, 9:43 p.m., Ashutosh Chauhan wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Lines 2425 (patched)
> > 
> >
> > Currently it will fetch all tables (acid or not). Is that intentional?

Yes


> On May 23, 2018, 9:43 p.m., Ashutosh Chauhan wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
> > Lines 1642 (patched)
> > 
> >
> > Return value should be List>

Same as above... it's ok for Hive to use fq table name everywhere as a string, 
so it should be ok here


> On May 23, 2018, 9:43 p.m., Ashutosh Chauhan wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> > Lines 738 (patched)
> > 
> >
> > I think this timeunit should be in minutes since this task need to run 
> > more frequently than that. That will make this config less error-prone.
> > Default value : 1 hour.

Changed the setting. Note that this wait is the wait until we check the next 
time if the check was a noop... I don't think it's a good idea to wait for an 
hour.
It will produce counterintuitive behavior where if tables are updated steadily, 
after each non-noop run it will find yet another table to update and do the 
full check, but one small gap will result at nothing being updated for an hour.
This is a throttle for checks, not analyze queries. We may add separate one for 
queries if desired.


- Sergey


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67125/#review203694
---


On May 15, 2018, 4:55 a.m., Sergey Shelukhin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67125/
> ---
> 
> (Updated May 15, 2018, 4:55 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Seong (Steve) Yeom.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> see jira. This should eventually integrate with ACID stats to determine what 
> stats are out of date, when that is done. Probably in separate jira if this 
> goes in first.
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  3d6fda6bd4 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverUtils.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 89129f99fe 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> b698c84080 
>   ql/src/test/org/apache/hadoop/hive/ql/stats/TestStatsUpdaterThread.java 
> PRE-CREATION 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
>  0be0aaa10c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  92d2e3f368 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  48f77b9878 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  264fdb9db9 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
>  ce7d2861dd 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  b223920e82 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/EnumValidator.java
>  PRE-CREATION 
>   
>

[jira] [Created] (HIVE-19720) backport multiple ACID and MM jiras to branch-3

2018-05-25 Thread Sergey Shelukhin (JIRA)

Sergey Shelukhin created HIVE-19720:
---

 Summary: backport multiple ACID and MM jiras to branch-3
 Key: HIVE-19720
 URL: https://issues.apache.org/jira/browse/HIVE-19720
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


To avoid chained test runs of branch-3 backporting one by one, I will run 
HiveQA on an epic combined patch, then commit patches w/proper commit 
separation via cherry-pick



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67263: HIVE-19602

2018-05-25 Thread Bharathkrishna Guruvayoor Murali via Review Board



> On May 25, 2018, 4:43 p.m., Sahil Takiar wrote:
> > are there any logic changes, or is most of the code just copied into the 
> > new class?

Most of the code is copied to new class. Mostly I have the doubts mentioned in 
above comments.


- Bharathkrishna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67263/#review203891
---


On May 23, 2018, 5:32 a.m., Bharathkrishna Guruvayoor Murali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67263/
> ---
> 
> (Updated May 23, 2018, 5:32 a.m.)
> 
> 
> Review request for hive, Sahil Takiar and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19602
> https://issues.apache.org/jira/browse/HIVE-19602
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Refactor inplace progress code in Hive-on-spark progress monitor to use 
> ProgressMonitor instance
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
> 7afd8864075aa0d9708274eea8839c662324c732 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkProgressMonitor.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67263/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bharathkrishna Guruvayoor Murali
> 
>

[jira] [Created] (HIVE-19719) Adding metastore batch API for partitions

2018-05-25 Thread Alexander Kolbasov (JIRA)

Alexander Kolbasov created HIVE-19719:
-

 Summary: Adding metastore batch API for partitions
 Key: HIVE-19719
 URL: https://issues.apache.org/jira/browse/HIVE-19719
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Affects Versions: 3.1.0, 4.0.0
Reporter: Alexander Kolbasov


Hive Metastore provides APIs for fetching a collection of objects (usually 
tables or partitions). These APIs provide a way to fetch all available objects 
so the size of the response is O(N) where N is the number of objects. These 
calls have several problems:

* All objects (and there may be thousands or even millions) should be fetched 
from the database, serialized to Java list of thrift objects then serialized 
into byte array for sending over the network. This creates spikes of huge 
memory pressure, especially since in some cases multiple of copies of the same 
data are present in memory (e.g. unserialized and serialized versions).
* Even though HMS tries to avoid string duplication by use of string interning 
in JAVA, duplicated strings must be serialized in the output array.
* Java has 2Gb limit on the maximum size of byte array, and crashes with Out Of 
Memory exception if this array size is exceeded
* Fetching huge amount of objects blows up DB caches and memory caches in the 
system.
Receiving such huge messages also creates memory pressure on the receiver side 
(usually HS2) which can cause it crashing with Out of Memory exception as well.
* Such requests have very big latencies since the server must collect all 
objects, serialize them and send them all to the network before the client can 
do anything with the result.

To prevent cases of Out Of Memory exceptions, the server now has a configurable 
limit on the maximum number of objects returned. This helps to avoid crashes, 
but doesn’t allow for correct query execution since the result will include 
random and incomplete set of K objects.

Currently this is addressed on the client side by simulating batching by 
getting list of table or partition names first and then requesting table 
information for parts of this list. Still, the list of objects can be big as 
well and this method requires locking to ensure that objects are not added or 
removed between the calls, especially if this is done outside of HS2.

Instead we can do simple modification of existing APIs which allows for batch 
iterator-style operations without keeping any server-side state. The main idea 
is to have a unique incrementing IDs for each objects. The IDs should be only 
unique within their container (e.g. table IDs should be unique within a 
database and partition IDs should be unique within a table). 
Such ID can be easily generated using database auto-increment mechanism or we 
can be simply reuse existing ID column that is already maintained by the Data 
Nucleus.
The request is then modified to include

* Starting ID i0
* Batch size (B)

The server fetches up to B objects starting from i0, serlalizes them and sends 
to the client. The client then requests next batch by using the ID of the last 
received request plus one. It is possible to construct an SQL query (either by 
using DataNucleus JDOQL or in DirectSQL code) which only selects needed objects 
avoiding big reads from the database. The client then iterates until it fetches 
all the objects and each request memory size is limited by the value of batch 
size.
If we extend the API a little bit, providing a way to get the minimum 
and maximum ID values (either via a separate call or piggybacked to the normal 
reply), clients can request such batches concurrently, thus also reducing the 
latency. Clients can easily estimate number of batches by knowing the total 
number of IDs. While this isn’t a precise method it is good enough to divide 
the work.

It is also possible to wrap this in a way similar to {{PartitionIterator}} and 
async-fetch next batch while we are processing current batch.

* Consistency considerations*
* 
HMS only provides consistency guarantees for a single call. The set of objects 
that should be returned may change while we are iterating over it. In some 
cases this is not an issue since HS2 may use ZooKeeper locks on the table to 
prevent modifications, but in some cases this may be an issue (for example for 
calls that originate from external systems. We should consider additions and 
removals separately.

* New objects are added during iteration. All new objects are always added at 
the ‘end’ of ID space, so they will be always picked up by the iterator. We 
assume that IDs are always incrementing.
* Some objects are removed during iteration. Removal of objects that are not 
already consumed is not a problem.  It is possible that some objects which were 
already consumed are returned. Although this results in an inconsistent list of 
objects, this situation is

[jira] [Created] (HIVE-19718) Adding partitions in bulk also fetches table for each partition

2018-05-25 Thread Alexander Kolbasov (JIRA)

Alexander Kolbasov created HIVE-19718:
-

 Summary: Adding partitions in bulk also fetches table for each 
partition
 Key: HIVE-19718
 URL: https://issues.apache.org/jira/browse/HIVE-19718
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 3.0.0
 Environment: Looking at {{convertToMPart}}:

{code:Java}
  private MPartition convertToMPart(Partition part, boolean useTableCD)
  throws InvalidObjectException, MetaException {
MTable mt = getMTable(part.getCatName(), part.getDbName(), 
part.getTableName());
...
{code}

So what we have as a result is that we fetch table for every partition where it 
should be done just once.
Reporter: Alexander Kolbasov


The ObjectStore.addPartitions() method does this:
{code:java}
for (Partition part : parts) {
  if (!part.getTableName().equals(tblName) || !part.getDbName().equals(dbName)) 
{
throw new MetaException("Partition does not belong to target table "
+ dbName + "." + tblName + ": " + part);
  }
  MPartition mpart = convertToMPart(part, true); // <-- Here
  toPersist.add(mpart);
  ...{code}





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67329: HIVE-19629: Enable Decimal64 reader after orc version upgrade

2018-05-25 Thread j . prasanth . j



> On May 25, 2018, 10:58 p.m., Prasanth_J wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
> > Lines 4402 (patched)
> > 
> >
> > actually this is unused variable.. i left a note to fix some test case. 
> > will fixup this parent and child data type variation mismatch in next patch.

(d + 1) % (d / 2) where d is DECIMAL_64 exhibits this behaviour. Mod and Divide 
does not support DECIMAL_64 whereas Addition does. So this expression becomes 
(+: DECIMAL_64) (%: NONE) (/: NONE). Mod is parent whose output variation is 
NONE but one of its child (+) has DECIMAL_64 variation which should be casted 
up to avoid runtime ClassCastException.


- Prasanth_J


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67329/#review203920
---


On May 25, 2018, 8:25 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67329/
> ---
> 
> (Updated May 25, 2018, 8:25 p.m.)
> 
> 
> Review request for hive and Matt McCline.
> 
> 
> Bugs: HIVE-19629
> https://issues.apache.org/jira/browse/HIVE-19629
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19629: Enable Decimal64 reader after orc version upgrade
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 931533a 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 0af91bd 
>   itests/src/test/resources/testconfiguration.properties d146f92 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  6d29163 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/GenericColumnVectorProducer.java
>  7af1b05 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  feccb87 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  4033b37 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
>  1cfe929 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/VectorDeserializeOrcWriter.java
>  de19b1d 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ConsumerFileMetadata.java
>  bf139c0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
>  0012afb 
>   pom.xml e48974b 
>   ql/pom.xml 06124f7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 2246901 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedInputFormatInterface.java
>  e74b185 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatchCtx.java 
> 6588385 
>   ql/src/java/org/apache/hadoop/hive/ql/io/NullRowsInputFormat.java e632d43 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java f461364 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> 8c7c72e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java 7485e60 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java 1a6db1f 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java 5b001a0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  d2e1a68 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
> c581bba 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 71682af 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  646b214 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
>  ed6d577 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 394f826 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java af43b14 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> fb2335a 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ef678a8 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java 
> d8a7af8 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java 1533ffa 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedORCReader.java 
> 0c9c95d 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
>  e478371 
>   ql/src/test/queries/clientpositive/llap_acid2.q a409c26 
>   ql/src/test/queries/clientpositive/llap_decimal64_reader.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/llap_uncompressed.q 875356c 
>   ql/src/test/results/clientpositive/acid_mapjoin.q.out 76a781e 
>   ql/src/test/results/clientpositive/acid_nullscan.q.out 6dad497 
>

Re: Review Request 67263: HIVE-19602

2018-05-25 Thread Bharathkrishna Guruvayoor Murali via Review Board



> On May 25, 2018, 4:43 p.m., Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java
> > Line 70 (original), 67 (patched)
> > 
> >
> > is this still used?

Will remove this if we do not need to pass the headers , footers etc.. to 
ProgressMonitor.
(ie. if the progress bar format shown in below comment is acceptable).


> On May 25, 2018, 4:43 p.m., Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java
> > Line 304 (original), 231 (patched)
> > 
> >
> > whats the point of this class?

I used this class to follow same pattern as in tez. I will add the logic 
similar to RenderStrategy used in tez while adding beeline progress bar, so 
this should be useful.


> On May 25, 2018, 4:43 p.m., Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkProgressMonitor.java
> > Lines 27 (patched)
> > 
> >
> > whats the impact of having an extra argument? does the formatting 
> > change at all?

It changes a bit,like this:

--
  STAGES   ATTEMPTSTATUS  TOTAL  COMPLETED  RUNNING  PENDING  
FAILED
--
Stage-1  0  FINISHED  1  100
   0
--
STAGES: 01/01[==>>] 100%  ELAPSED TIME: 1.01 s
--

Notice the bit of extra space at the end. But other than that, it looks pretty 
much same.


- Bharathkrishna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67263/#review203891
---


On May 23, 2018, 5:32 a.m., Bharathkrishna Guruvayoor Murali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67263/
> ---
> 
> (Updated May 23, 2018, 5:32 a.m.)
> 
> 
> Review request for hive, Sahil Takiar and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19602
> https://issues.apache.org/jira/browse/HIVE-19602
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Refactor inplace progress code in Hive-on-spark progress monitor to use 
> ProgressMonitor instance
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
> 7afd8864075aa0d9708274eea8839c662324c732 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkProgressMonitor.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67263/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bharathkrishna Guruvayoor Murali
> 
>

Re: Review Request 67329: HIVE-19629: Enable Decimal64 reader after orc version upgrade

2018-05-25 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67329/#review203920
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
Lines 4402 (patched)


actually this is unused variable.. i left a note to fix some test case. 
will fixup this parent and child data type variation mismatch in next patch.


- Prasanth_J


On May 25, 2018, 8:25 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67329/
> ---
> 
> (Updated May 25, 2018, 8:25 p.m.)
> 
> 
> Review request for hive and Matt McCline.
> 
> 
> Bugs: HIVE-19629
> https://issues.apache.org/jira/browse/HIVE-19629
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19629: Enable Decimal64 reader after orc version upgrade
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 931533a 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 0af91bd 
>   itests/src/test/resources/testconfiguration.properties d146f92 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  6d29163 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/GenericColumnVectorProducer.java
>  7af1b05 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  feccb87 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  4033b37 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
>  1cfe929 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/VectorDeserializeOrcWriter.java
>  de19b1d 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ConsumerFileMetadata.java
>  bf139c0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
>  0012afb 
>   pom.xml e48974b 
>   ql/pom.xml 06124f7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 2246901 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedInputFormatInterface.java
>  e74b185 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatchCtx.java 
> 6588385 
>   ql/src/java/org/apache/hadoop/hive/ql/io/NullRowsInputFormat.java e632d43 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java f461364 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> 8c7c72e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java 7485e60 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java 1a6db1f 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java 5b001a0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  d2e1a68 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
> c581bba 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 71682af 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  646b214 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
>  ed6d577 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 394f826 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java af43b14 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> fb2335a 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ef678a8 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java 
> d8a7af8 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java 1533ffa 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedORCReader.java 
> 0c9c95d 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
>  e478371 
>   ql/src/test/queries/clientpositive/llap_acid2.q a409c26 
>   ql/src/test/queries/clientpositive/llap_decimal64_reader.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/llap_uncompressed.q 875356c 
>   ql/src/test/results/clientpositive/acid_mapjoin.q.out 76a781e 
>   ql/src/test/results/clientpositive/acid_nullscan.q.out 6dad497 
>   ql/src/test/results/clientpositive/acid_table_stats.q.out 2596922 
>   ql/src/test/results/clientpositive/annotate_stats_part.q.out 9e45101 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out b502957 
>   ql/src/test/results/clientpositive/autoColumnStats_4.q.out 5be906e 
>   
> ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
>  a717b80 
>   
> ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
> fa58add

Re: Review Request 67329: HIVE-19629: Enable Decimal64 reader after orc version upgrade

2018-05-25 Thread Matt McCline


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67329/#review203919
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
Lines 4402 (patched)


NOTE TO SELF: Look at this again.


- Matt McCline


On May 25, 2018, 8:25 p.m., Prasanth_J wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67329/
> ---
> 
> (Updated May 25, 2018, 8:25 p.m.)
> 
> 
> Review request for hive and Matt McCline.
> 
> 
> Bugs: HIVE-19629
> https://issues.apache.org/jira/browse/HIVE-19629
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-19629: Enable Decimal64 reader after orc version upgrade
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 931533a 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
> 0af91bd 
>   itests/src/test/resources/testconfiguration.properties d146f92 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
>  6d29163 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/GenericColumnVectorProducer.java
>  7af1b05 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
>  feccb87 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
>  4033b37 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
>  1cfe929 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/VectorDeserializeOrcWriter.java
>  de19b1d 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ConsumerFileMetadata.java
>  bf139c0 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
>  0012afb 
>   pom.xml e48974b 
>   ql/pom.xml 06124f7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 2246901 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedInputFormatInterface.java
>  e74b185 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatchCtx.java 
> 6588385 
>   ql/src/java/org/apache/hadoop/hive/ql/io/NullRowsInputFormat.java e632d43 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java f461364 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> 8c7c72e 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java 7485e60 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java 1a6db1f 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java 5b001a0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  d2e1a68 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
> c581bba 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 71682af 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
>  646b214 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java
>  ed6d577 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
> 394f826 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java af43b14 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> fb2335a 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ef678a8 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java 
> d8a7af8 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java 1533ffa 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedORCReader.java 
> 0c9c95d 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
>  e478371 
>   ql/src/test/queries/clientpositive/llap_acid2.q a409c26 
>   ql/src/test/queries/clientpositive/llap_decimal64_reader.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/llap_uncompressed.q 875356c 
>   ql/src/test/results/clientpositive/acid_mapjoin.q.out 76a781e 
>   ql/src/test/results/clientpositive/acid_nullscan.q.out 6dad497 
>   ql/src/test/results/clientpositive/acid_table_stats.q.out 2596922 
>   ql/src/test/results/clientpositive/annotate_stats_part.q.out 9e45101 
>   ql/src/test/results/clientpositive/annotate_stats_table.q.out b502957 
>   ql/src/test/results/clientpositive/autoColumnStats_4.q.out 5be906e 
>   
> ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
>  a717b80 
>   
> ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
> fa58add 
>   ql/src/test/results/clientpositive/deleteAnalyze.q.out d98114b 
>

Re: Review Request 67329: HIVE-19629: Enable Decimal64 reader after orc version upgrade

2018-05-25 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67329/
---

(Updated May 25, 2018, 8:24 p.m.)


Review request for hive.


Bugs: HIVE-19629
https://issues.apache.org/jira/browse/HIVE-19629


Repository: hive-git


Description
---

HIVE-19629: Enable Decimal64 reader after orc version upgrade


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 931533a 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
0af91bd 
  itests/src/test/resources/testconfiguration.properties d146f92 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
 6d29163 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/GenericColumnVectorProducer.java
 7af1b05 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 feccb87 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 4033b37 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 1cfe929 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/VectorDeserializeOrcWriter.java
 de19b1d 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ConsumerFileMetadata.java
 bf139c0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
 0012afb 
  pom.xml e48974b 
  ql/pom.xml 06124f7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 2246901 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedInputFormatInterface.java
 e74b185 
  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatchCtx.java 
6588385 
  ql/src/java/org/apache/hadoop/hive/ql/io/NullRowsInputFormat.java e632d43 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java f461364 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 8c7c72e 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java 7485e60 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java 1a6db1f 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java 5b001a0 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
 d2e1a68 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 
c581bba 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 71682af 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
 646b214 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java 
ed6d577 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 
394f826 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java af43b14 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
fb2335a 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ef678a8 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java 
d8a7af8 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java 1533ffa 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedORCReader.java 
0c9c95d 
  
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestVectorizedOrcAcidRowBatchReader.java
 e478371 
  ql/src/test/queries/clientpositive/llap_acid2.q a409c26 
  ql/src/test/queries/clientpositive/llap_decimal64_reader.q PRE-CREATION 
  ql/src/test/queries/clientpositive/llap_uncompressed.q 875356c 
  ql/src/test/results/clientpositive/acid_mapjoin.q.out 76a781e 
  ql/src/test/results/clientpositive/acid_nullscan.q.out 6dad497 
  ql/src/test/results/clientpositive/acid_table_stats.q.out 2596922 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 9e45101 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out b502957 
  ql/src/test/results/clientpositive/autoColumnStats_4.q.out 5be906e 
  
ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
 a717b80 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
fa58add 
  ql/src/test/results/clientpositive/deleteAnalyze.q.out d98114b 
  ql/src/test/results/clientpositive/druid/druidmini_mv.q.out 35f4c1b 
  ql/src/test/results/clientpositive/extrapolate_part_stats_date.q.out f1cd05c 
  ql/src/test/results/clientpositive/extrapolate_part_stats_full.q.out bec6dd4 
  ql/src/test/results/clientpositive/extrapolate_part_stats_partial.q.out 
1a13b21 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out c89c22c 
  ql/src/test/results/clientpositive/llap/acid_vectorization_original.q.out 
38a9770 
  ql/src/test/results/clientpositive/llap/alter_merge_orc.q.out 7b33e8e 
  ql/src/test/results/clientpositive/llap/alter_merge_stats_orc.q.out c8b831d 
  ql/src/test/results/clientpositive/llap/column_table_stats_orc.q.out 11a9c0e

Review Request 67329: HIVE-19629: Enable Decimal64 reader after orc version upgrade

2018-05-25 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67329/
---

Review request for hive.


Bugs: HIVE-19629
https://issues.apache.org/jira/browse/HIVE-19629


Repository: hive-git


Description
---

HIVE-19629: Enable Decimal64 reader after orc version upgrade


Diffs
-

  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 
0af91bda95b49d87aac186af020d9f3f301c4393 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 4033b379defbef2ed952bee7e4f737149a6c5a9d 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 1cfe92978a029ef68aa1af6eeda5a5aacbad255e 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/ConsumerFileMetadata.java
 bf139c071ccee07be3094ab64b9efafcb95c7b7c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileMetadata.java
 0012afb3efa1fa67c8c897711ab102c833f1f0a0 
  pom.xml e48974bab4906d2fa5e81d048c596e084e43205a 
  ql/pom.xml 06124f738779141e97e67f2e07c1d13dd85425d0 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 
f4613647314a1b70b6cde3005ddce9d61b154de4 
  
ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedTreeReaderFactory.java
 646b21424960e56b90e1320e137c6b7cd8143c52 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java 
af43b14abfc474dd59646b02e31a792a04ec86d6 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
fb2335ab1f990ca8f731192afd016506287473c1 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java 
ef678a8eb323149ee11cbe77c5b55a7c651cf474 
  ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcSerDeStats.java 
1533ffae2a84ca4c71168f744db90fd732ecba04 
  ql/src/test/results/clientpositive/acid_mapjoin.q.out 
76a781e94e11a845b1143962d820c70b72214d4b 
  ql/src/test/results/clientpositive/acid_nullscan.q.out 
6dad4974ae42fbab665a2968e38e37bcc0be40d7 
  ql/src/test/results/clientpositive/acid_table_stats.q.out 
2596922fff380746d5952e52e0922271b0968f5e 
  ql/src/test/results/clientpositive/annotate_stats_part.q.out 
9e45101fe6cf6927dfec3a2b7dccb42fbfadb5d8 
  ql/src/test/results/clientpositive/annotate_stats_table.q.out 
b502957e96e0faa68147059a3ad105a9d02a528a 
  ql/src/test/results/clientpositive/autoColumnStats_4.q.out 
5be906eee456c27953454d9f2326f6cf67c4e3c1 
  
ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out
 a717b80d045ecbcb9066c02fa1dae4985be36e9d 
  ql/src/test/results/clientpositive/columnStatsUpdateForStatsOptimizer_2.q.out 
fa58add84d720ad866a4df31a637ecc9ad8a8cdb 
  ql/src/test/results/clientpositive/deleteAnalyze.q.out 
d98114bff19d668a64befb888601eef548e48bc3 
  ql/src/test/results/clientpositive/druid/druidmini_mv.q.out 
35f4c1b50b58c83534597fca43074390cb20f38d 
  ql/src/test/results/clientpositive/extrapolate_part_stats_date.q.out 
f1cd05cb9bc00f8fc97eec97fa76cd3697aa7f25 
  ql/src/test/results/clientpositive/extrapolate_part_stats_full.q.out 
bec6dd4a0ee80304e4b6e3ad8e12a37d5365ce0d 
  ql/src/test/results/clientpositive/extrapolate_part_stats_partial.q.out 
1a13b21e4e87908ec83a80f482707bc651627ef1 
  ql/src/test/results/clientpositive/llap/acid_bucket_pruning.q.out 
c89c22cebaed12fef8f0c0caaf0e641887a37d2a 
  ql/src/test/results/clientpositive/llap/acid_vectorization_original.q.out 
38a97700e67b99e86d1a0b9af67693482cf76f65 
  ql/src/test/results/clientpositive/llap/alter_merge_orc.q.out 
7b33e8e93d1561205fce8fae4ad539ce8f875b8f 
  ql/src/test/results/clientpositive/llap/alter_merge_stats_orc.q.out 
c8b831df7975bd7267d2c2b07f08e097493414ed 
  ql/src/test/results/clientpositive/llap/column_table_stats_orc.q.out 
11a9c0ecbe636f24b67a6aa0c0c021da07529730 
  ql/src/test/results/clientpositive/llap/default_constraint.q.out 
cb69af7579543890c42702e09a5808396870f3f5 
  ql/src/test/results/clientpositive/llap/deleteAnalyze.q.out 
a125fd62290957b31a265c24199cdff8b73bdbb7 
  ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out 
22f2860cfa5571b43fc9d37f149fdef6dcb40556 
  ql/src/test/results/clientpositive/llap/dynamic_semijoin_user_level.q.out 
048712eed1175c468f0002387d78fd7e36028b1d 
  ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out 
709100f1182053e4bec5fa694534f52387e1b789 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization2.q.out 
72f3b8bf63d06cd99b66653a76240387bfa89a9d 
  ql/src/test/results/clientpositive/llap/dynpart_sort_optimization_acid.q.out 
fa51dd3e27dd7cc82ec95c470d3f9ac73d28b3d7 
  ql/src/test/results/clientpositive/llap/enforce_constraint_notnull.q.out 
5a9d263267816bb21f2836265bc5b65cf6b84ffd 
  
ql/src/test/results/clientpositive/llap/extrapolate_part_stats_partial_ndv.q.out
 8a0da7d73d602f9cc0f86c1ebc935cab059ae000

[jira] [Created] (HIVE-19717) Dummy jira to run tests on branch-3

2018-05-25 Thread Vineet Garg (JIRA)

Vineet Garg created HIVE-19717:
--

 Summary: Dummy jira to run tests on branch-3
 Key: HIVE-19717
 URL: https://issues.apache.org/jira/browse/HIVE-19717
 Project: Hive
  Issue Type: Task
Reporter: Vineet Garg
Assignee: Vineet Garg






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19716) Set spark.local.dir for a few more HoS integration tests

2018-05-25 Thread Sahil Takiar (JIRA)

Sahil Takiar created HIVE-19716:
---

 Summary: Set spark.local.dir for a few more HoS integration tests
 Key: HIVE-19716
 URL: https://issues.apache.org/jira/browse/HIVE-19716
 Project: Hive
  Issue Type: Test
  Components: Spark
Reporter: Sahil Takiar
Assignee: Sahil Takiar


There are a few more flaky tests that are failing because the run a HoS queries 
that writes some temp data to {{/tmp/}}. These tests are regular JUnit tests, 
so they weren't covered in the previous attempts to do this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19715) Consolidated and flexible API for fetching partition metadata from HMS

2018-05-25 Thread Todd Lipcon (JIRA)

Todd Lipcon created HIVE-19715:
--

 Summary: Consolidated and flexible API for fetching partition 
metadata from HMS
 Key: HIVE-19715
 URL: https://issues.apache.org/jira/browse/HIVE-19715
 Project: Hive
  Issue Type: New Feature
  Components: Standalone Metastore
Reporter: Todd Lipcon


Currently, the HMS thrift API exposes 17 different APIs for fetching 
partition-related information. There is somewhat of a combinatorial explosion 
going on, where each API has variants with and without "auth" info, by pspecs 
vs names, by filters, by exprs, etc. Having all of these separate APIs long 
term is a maintenance burden and also more confusing for consumers.

Additionally, even with all of these APIs, there is a lack of granularity in 
fetching only the information needed for a particular use case. For example, in 
some use cases it may be beneficial to only fetch the partition locations 
without wasting effort fetching statistics, etc.

This JIRA proposes that we add a new "one API to rule them all" for fetching 
partition info. The request and response would be encapsulated in structs. Some 
desirable properties:
- the request should be able to specify which pieces of information are 
required (eg location, properties, etc)
- in the case of partition parameters, the request should be able to do either 
whitelisting or blacklisting (eg to exclude large incremental column stats HLL 
dumped in there by Impala)
- the request should optionally specify auth info (to encompas the "with_auth" 
variants)
- the request should be able to designate the set of partitions to access 
through one of several different methods (eg "all", list, expr, 
part_vals, etc) 
- the struct should be easily evolvable so that new pieces of info can be added
- the response should be designed in such a way as to avoid transferring 
redundant information for common cases (eg simple "dictionary coding" of 
strings like parameter names, etc)
- the API should support some form of pagination for tables with large 
partition counts




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19714) TransactionalValidationListene.conformToAcid() only checks table level StorageDescriptor

2018-05-25 Thread Eugene Koifman (JIRA)

Eugene Koifman created HIVE-19714:
-

 Summary: TransactionalValidationListene.conformToAcid() only 
checks table level StorageDescriptor
 Key: HIVE-19714
 URL: https://issues.apache.org/jira/browse/HIVE-19714
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 2.2.0
Reporter: Eugene Koifman


A table may actually have different SD for each partition so a proper check to 
for full CRUD table would check all of them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67263: HIVE-19602

2018-05-25 Thread Sahil Takiar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67263/#review203891
---



are there any logic changes, or is most of the code just copied into the new 
class?


ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java
Line 70 (original), 67 (patched)


is this still used?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java
Line 304 (original), 231 (patched)


whats the point of this class?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkProgressMonitor.java
Lines 27 (patched)


whats the impact of having an extra argument? does the formatting change at 
all?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkProgressMonitor.java
Lines 77 (patched)


the current states LGTM


- Sahil Takiar


On May 23, 2018, 5:32 a.m., Bharathkrishna Guruvayoor Murali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67263/
> ---
> 
> (Updated May 23, 2018, 5:32 a.m.)
> 
> 
> Review request for hive, Sahil Takiar and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19602
> https://issues.apache.org/jira/browse/HIVE-19602
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Refactor inplace progress code in Hive-on-spark progress monitor to use 
> ProgressMonitor instance
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
> 7afd8864075aa0d9708274eea8839c662324c732 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkProgressMonitor.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67263/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bharathkrishna Guruvayoor Murali
> 
>

[GitHub] hive pull request #357: Branch 1.2

2018-05-25 Thread ey1984

GitHub user ey1984 opened a pull request:

https://github.com/apache/hive/pull/357

Branch 1.2

Hello,

I'm using your hive-jdbc (1.2.1) as dependency for 2 applications deployed 
into 2 docker containers (Java Code and Python Code).
For both containers, it runs behind a proxy system when I deploy on 
qualification and production environment.
For java, (-Dhttp.proxyHost=MyProxyHost -Dhttp.proxyPort=MyProxyPort)
For python, I set HTTP_PROXY=http://myproxyhost:myproxyport

And after deployed, and launched, a timeout occurs when I want to hit the 
hive server. I call hive by url jdbc://hive

So after debugging your source code, I added some code (PR as requested) in 
order to get the proxy system (env from os or jvm configuration) and It works 
fine for both containers.

Is this correction acceptable or is there any other solution to hit Hive 
Server by using proxy ?

Python : I use jaydepbeapi and Java : Only 
DriverManager.getConnection("jdbc://hive2)

Thanks a lot

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ey1984/hive branch-1.2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/357.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #357


commit 2900b5c687f7a2e11bf1b56ddc17aa271e557ff2
Author: ey1984 
Date:   2018-04-30T21:31:39Z

Get proxy system

commit b07aa6a4d228a40c9b45ff5d99a7104a5dbb8443
Author: ey1984 
Date:   2018-04-30T21:54:28Z

merge conflict caused by PR Hive-Proxy-System

commit 3962b9b3b54f5e07854331d989911b645ef344bd
Author: Ekrem YILMAZ 
Date:   2018-05-14T12:06:21Z

Get Proxy Settings By UseSystemProperties

Get Proxy Settings by UseSystemProperties and keep getEnv

commit 2e98af05005161eb33c255022d0487ddd808dfb0
Author: Ekrem YILMAZ 
Date:   2018-05-14T12:09:36Z

Merge branch 'HIVE-Proxy-System' into branch-1.2

commit 59860ca463eec571421d875ef1ed32f545ecfef9
Author: Ekrem YILMAZ 
Date:   2018-05-14T12:10:08Z

Merge with Hive-Proxy-System

commit 913c8c8f620c62316d89d7b9187bfcad3b700f16
Author: Ekrem YILMAZ 
Date:   2018-05-14T12:11:47Z

Merge with Hive-Proxy-System




---

[GitHub] hive pull request #351: HIVE - Get proxy system

2018-05-25 Thread ey1984

Github user ey1984 closed the pull request at:

https://github.com/apache/hive/pull/351


---

[GitHub] hive pull request #356: Hive proxy system

2018-05-25 Thread ey1984

Github user ey1984 closed the pull request at:

https://github.com/apache/hive/pull/356


---

[jira] [Created] (HIVE-19713) itests/hive-jmh should reference not reference a conctrete storage-api version

2018-05-25 Thread Zoltan Haindrich (JIRA)

Zoltan Haindrich created HIVE-19713:
---

 Summary: itests/hive-jmh should reference not reference a 
conctrete storage-api version
 Key: HIVE-19713
 URL: https://issues.apache.org/jira/browse/HIVE-19713
 Project: Hive
  Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


this is a bigger problem on branch-3; where storage-api is 2.6.1; but hive-jmh 
references 2.7.0 (which is for master)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] hive pull request #356: Hive proxy system

2018-05-25 Thread ey1984

GitHub user ey1984 reopened a pull request:

https://github.com/apache/hive/pull/356

Hive proxy system

Hello,

I'm using your hive-jdbc (1.2.1) as dependency for 2 applications deployed 
into 2 docker containers (Java Code and Python Code).
For both containers, it runs behind a proxy system when I deploy on 
qualification and production environment.
For java, (-Dhttp.proxyHost=MyProxyHost -Dhttp.proxyPort=MyProxyPort)
For python, I set HTTP_PROXY=http://myproxyhost:myproxyport

And after deployed, and launched, a timeout occurs when I want to hit the 
hive server. I call hive by url jdbc://hive

So after debugging your source code, I added some code (PR as requested) in 
order to get the proxy system (env from os or jvm configuration) and It works 
fine for both containers.

Is this correction acceptable or is there any other solution to hit Hive 
Server by using proxy ?

Python : I use jaydepbeapi and Java : Only 
DriverManager.getConnection("jdbc://hive2)

Thanks a lot

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ey1984/hive HIVE-Proxy-System

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/356.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #356


commit 2900b5c687f7a2e11bf1b56ddc17aa271e557ff2
Author: ey1984 
Date:   2018-04-30T21:31:39Z

Get proxy system

commit 3962b9b3b54f5e07854331d989911b645ef344bd
Author: Ekrem YILMAZ 
Date:   2018-05-14T12:06:21Z

Get Proxy Settings By UseSystemProperties

Get Proxy Settings by UseSystemProperties and keep getEnv




---

[GitHub] hive pull request #356: Hive proxy system

2018-05-25 Thread ey1984

Github user ey1984 closed the pull request at:

https://github.com/apache/hive/pull/356


---

[GitHub] hive pull request #356: Hive proxy system

2018-05-25 Thread ey1984

GitHub user ey1984 opened a pull request:

https://github.com/apache/hive/pull/356

Hive proxy system

Hello,

I'm using your hive-jdbc (1.2.1) as dependency for 2 applications deployed 
into 2 docker containers (Java Code and Python Code).
For both containers, it runs behind a proxy system when I deploy on 
qualification and production environment.
For java, (-Dhttp.proxyHost=MyProxyHost -Dhttp.proxyPort=MyProxyPort)
For python, I set HTTP_PROXY=http://myproxyhost:myproxyport

And after deployed, and launched, a timeout occurs when I want to hit the 
hive server. I call hive by url jdbc://hive

So after debugging your source code, I added some code (PR as requested) in 
order to get the proxy system (env from os or jvm configuration) and It works 
fine for both containers.

Is this correction acceptable or is there any other solution to hit Hive 
Server by using proxy ?

Python : I use jaydepbeapi and Java : Only 
DriverManager.getConnection("jdbc://hive2)

Thanks a lot

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ey1984/hive HIVE-Proxy-System

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/356.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #356


commit 2900b5c687f7a2e11bf1b56ddc17aa271e557ff2
Author: ey1984 
Date:   2018-04-30T21:31:39Z

Get proxy system

commit 3962b9b3b54f5e07854331d989911b645ef344bd
Author: Ekrem YILMAZ 
Date:   2018-05-14T12:06:21Z

Get Proxy Settings By UseSystemProperties

Get Proxy Settings by UseSystemProperties and keep getEnv




---

[GitHub] hive pull request #338: Get proxy system

2018-05-25 Thread ey1984

Github user ey1984 closed the pull request at:

https://github.com/apache/hive/pull/338


---

[jira] [Created] (HIVE-19712) In 3 valued logic mode (b and not b) may not be simplified to false

2018-05-25 Thread Zoltan Haindrich (JIRA)

Zoltan Haindrich created HIVE-19712:
---

 Summary: In 3 valued logic mode (b and not b) may not be 
simplified to false
 Key: HIVE-19712
 URL: https://issues.apache.org/jira/browse/HIVE-19712
 Project: Hive
  Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


currently its simplified to false; but that is not correct when a is unknown




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19711) Refactor Hive Schema Tool

2018-05-25 Thread Miklos Gergely (JIRA)

Miklos Gergely created HIVE-19711:
-

 Summary: Refactor Hive Schema Tool
 Key: HIVE-19711
 URL: https://issues.apache.org/jira/browse/HIVE-19711
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Miklos Gergely
Assignee: Miklos Gergely
 Fix For: 3.0.0


HiveSchemaTool is an 1500 lines long class trying to do everything It shold be 
cut into multiple classes doing smaller components.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19710) Hive Schema Tool createCatalog command is not working with postgres DB

2018-05-25 Thread Miklos Gergely (JIRA)

Miklos Gergely created HIVE-19710:
-

 Summary: Hive Schema Tool createCatalog command is not working 
with postgres DB
 Key: HIVE-19710
 URL: https://issues.apache.org/jira/browse/HIVE-19710
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.0.0
Reporter: Miklos Gergely
Assignee: Miklos Gergely
 Fix For: 3.0.0


Hive Schema Tool can't create catalogs if the db is postgres due to trying to 
run an aggregated query in FOR UPDATE mode which is not supported.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67125: HIVE-19418 add background stats updater similar to compactor

Re: Review Request 67125: HIVE-19418 add background stats updater similar to compactor

[jira] [Created] (HIVE-19720) backport multiple ACID and MM jiras to branch-3

Re: Review Request 67263: HIVE-19602

[jira] [Created] (HIVE-19719) Adding metastore batch API for partitions

[jira] [Created] (HIVE-19718) Adding partitions in bulk also fetches table for each partition

Re: Review Request 67329: HIVE-19629: Enable Decimal64 reader after orc version upgrade

Re: Review Request 67263: HIVE-19602

Re: Review Request 67329: HIVE-19629: Enable Decimal64 reader after orc version upgrade

Re: Review Request 67329: HIVE-19629: Enable Decimal64 reader after orc version upgrade

Re: Review Request 67329: HIVE-19629: Enable Decimal64 reader after orc version upgrade

Review Request 67329: HIVE-19629: Enable Decimal64 reader after orc version upgrade

[jira] [Created] (HIVE-19717) Dummy jira to run tests on branch-3

[jira] [Created] (HIVE-19716) Set spark.local.dir for a few more HoS integration tests

[jira] [Created] (HIVE-19715) Consolidated and flexible API for fetching partition metadata from HMS

[jira] [Created] (HIVE-19714) TransactionalValidationListene.conformToAcid() only checks table level StorageDescriptor

Re: Review Request 67263: HIVE-19602

[GitHub] hive pull request #357: Branch 1.2

[GitHub] hive pull request #351: HIVE - Get proxy system

[GitHub] hive pull request #356: Hive proxy system

[jira] [Created] (HIVE-19713) itests/hive-jmh should reference not reference a conctrete storage-api version

[GitHub] hive pull request #356: Hive proxy system

[GitHub] hive pull request #356: Hive proxy system

[GitHub] hive pull request #356: Hive proxy system

[GitHub] hive pull request #338: Get proxy system

[jira] [Created] (HIVE-19712) In 3 valued logic mode (b and not b) may not be simplified to false

[jira] [Created] (HIVE-19711) Refactor Hive Schema Tool

[jira] [Created] (HIVE-19710) Hive Schema Tool createCatalog command is not working with postgres DB

28 matches

Site Navigation

Mail list logo

Footer information