Re: Review Request 67263: HIVE-19602

[Bharathkrishna Guruvayoor Murali via Review Board]

> On May 25, 2018, 4:43 p.m., Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java
> > Line 304 (original), 231 (patched)
> > 
> >
> > whats the point of this class?
> 
> Bharathkrishna Guruvayoor Murali wrote:
> I used this class to follow same pattern as in tez. I will add the logic 
> similar to RenderStrategy used in tez while adding beeline progress bar, so 
> this should be useful.
> 
> Sahil Takiar wrote:
> Ok, can this be a private class? Is it used outside `SparkJobMonitor`?
> 
> Bharathkrishna Guruvayoor Murali wrote:
> Not used outside SparkJobMonitor. Making private

I actually removed this class, because I was using the static class in a 
"non-static" kind of way. Just donot want to have code with wrong pattern even 
though it will bee changed soon.
I will add it when it is actually required, ie. later when I need to introduce 
progress bar in beeline.


- Bharathkrishna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67263/#review203891
---


On June 1, 2018, 4:58 a.m., Bharathkrishna Guruvayoor Murali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67263/
> ---
> 
> (Updated June 1, 2018, 4:58 a.m.)
> 
> 
> Review request for hive, Sahil Takiar and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19602
> https://issues.apache.org/jira/browse/HIVE-19602
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Refactor inplace progress code in Hive-on-spark progress monitor to use 
> ProgressMonitor instance
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
> 7afd8864075aa0d9708274eea8839c662324c732 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkProgressMonitor.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67263/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bharathkrishna Guruvayoor Murali
> 
>

Re: Review Request 67263: HIVE-19602

[Bharathkrishna Guruvayoor Murali via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67263/
---

(Updated June 1, 2018, 4:58 a.m.)


Review request for hive, Sahil Takiar and Vihang Karajgaonkar.


Bugs: HIVE-19602
https://issues.apache.org/jira/browse/HIVE-19602


Repository: hive-git


Description
---

Refactor inplace progress code in Hive-on-spark progress monitor to use 
ProgressMonitor instance


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
7afd8864075aa0d9708274eea8839c662324c732 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkProgressMonitor.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/67263/diff/3/

Changes: https://reviews.apache.org/r/67263/diff/2-3/


Testing
---


Thanks,

Bharathkrishna Guruvayoor Murali

[jira] [Created] (HIVE-19758) Set hadoop.version=3.1.0 in standalone-metastore

[Andrew Sherman (JIRA)]

Andrew Sherman created HIVE-19758:
-

 Summary: Set hadoop.version=3.1.0 in standalone-metastore
 Key: HIVE-19758
 URL: https://issues.apache.org/jira/browse/HIVE-19758
 Project: Hive
  Issue Type: Task
Reporter: Andrew Sherman
Assignee: Andrew Sherman


When HIVE-19243 set hadoop.version=3.1.0 it did not change the value used in 
standalone-metastore which still uses 3.0.0-beta1.
 At the moment standalone-metastore is still a module of hive and so this can 
suck in the wrong code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19757) hive.version.shortname should be 4.0

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-19757:
--

 Summary: hive.version.shortname should be 4.0
 Key: HIVE-19757
 URL: https://issues.apache.org/jira/browse/HIVE-19757
 Project: Hive
  Issue Type: Bug
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


pom.xml still points to 
{{3.1.0}} which causes issues 
with schemaTool init scripts



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67399: HIVE-19503 Create a test that checks for dropPartitions with directSql

[Vihang Karajgaonkar via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67399/#review204139
---



Thanks for the tests. Can we add a one more test which repeats the same test 
accross multiple sessions (can be simulated by using objectstore instances)? So 
for example: 

1. Session 1 : list partitions with jdo, Session 2 : list partitions with jdo
2. Session 1 : drop partitions with directSQL
3. Session 2 : list partitions with jdo


standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 2562 (patched)


why do we need it to be protected? Can we use package private instead?


- Vihang Karajgaonkar


On May 31, 2018, 1:53 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67399/
> ---
> 
> (Updated May 31, 2018, 1:53 p.m.)
> 
> 
> Review request for hive, Alexander Kolbasov and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19503
> https://issues.apache.org/jira/browse/HIVE-19503
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch contains 2 tests:
> 
> Test which checks if the JDO cache is able to handle directSql partition drops
> Test which checks if the directSQL partition drop removes every connected 
> data from the RDBMS tables.
> To create these tests we have 2 helper methods:
> 
> Method to create the partitioned table
> Method to check the number of rows in a given RDBMS table
> Added a new ObjectStore.dropPartitionsInternal method which only visible for 
> testing so we can make sure that the dropPartition is using directSql and 
> does not fall back to JDO.
> 
> Fixed a problem where some of the tables are not created automatically by the 
> tests, adding new init queries to MetaStoreDirectSql.ensureDbInit method
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  5bb1985 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  b15d89d 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  9912213 
> 
> 
> Diff: https://reviews.apache.org/r/67399/diff/1/
> 
> 
> Testing
> ---
> 
> Run the new tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Review Request 67406: HIVE-19663 refactor LLAP IO report generation

[Sergey Shelukhin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67406/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java 
f4a549c529 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/CacheContentsTracker.java
 6a361fa4fc 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java 
10a4bfbb58 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapOomDebugDump.java 
29ddf5c112 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 
af1b699771 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
5e102d93de 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheMemoryManager.java
 31cd45f103 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCachePolicy.java 
33236362b5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelFifoCachePolicy.java
 f7f80a8958 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelLrfuCachePolicy.java
 7787cb4867 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/MemoryManager.java 
65b1d4f654 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/SerDeLowLevelCacheImpl.java
 cb89d12e80 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleBufferManager.java 
a1b6caeb60 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
5003d9b5e5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/MetadataCache.java 
0184e3053f 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestBuddyAllocator.java 
714efef182 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestLowLevelCacheImpl.java
 b19cdcf531 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestOrcMetadataCache.java
 58c918c200 


Diff: https://reviews.apache.org/r/67406/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-19756) Insert request with UNION ALL and lateral view explode

[JIRA]

Frédéric ESCANDELL created HIVE-19756:
-

 Summary: Insert request with UNION ALL and lateral view explode
 Key: HIVE-19756
 URL: https://issues.apache.org/jira/browse/HIVE-19756
 Project: Hive
  Issue Type: Bug
 Environment: HDP 2.6.4
Reporter: Frédéric ESCANDELL


Hi,

While executing this code snippet, no data is inserted in the final table t3.

By replacing UNION ALL by UNION or the "lateral view explode" the code works 
properly.

 

 
{code:sql}
DROP table t1;
DROP table t2;
DROP table t3;


CREATE TABLE t1(cle string,valeur array>)
ROW FORMAT SERDE  'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS 
  INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
  OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat';

INSERT INTO table t1 select * from (select "a",array(named_struct('v','x'), 
named_struct('v','y'))) tmp;

 CREATE TABLE t2(cle string,valeur array>)
ROW FORMAT SERDE  'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS 
  INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
  OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat';

INSERT INTO table t2 select * from (select "b",array(named_struct('v','z'), 
named_struct('v','w'))) tmp;

DROP view v1;
DROP table t3;

CREATE VIEW v1 (cle,valeur) 
AS
select base.cle,val.v from (select cle,valeur from t1) as base
lateral view explode(base.valeur) a as val
union
select base1.cle,val.v from (select cle,valeur from t2) as base1
lateral view explode(base1.valeur) a as val;

 CREATE TABLE t3(cle string,valeur string)
ROW FORMAT SERDE  'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS 
  INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
  OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat';

insert into t3 
select * from v1;
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

REMINDER: Apache EU Roadshow 2018 in Berlin is less than 2 weeks away!

[sharan]

Hello Apache Supporters and Enthusiasts

This is a reminder that our Apache EU Roadshow in Berlin is less than 
two weeks away and we need your help to spread the word. Please let your 
work colleagues, friends and anyone interested in any attending know 
about our Apache EU Roadshow event.


We have a great schedule including tracks on Apache Tomcat, Apache Http 
Server, Microservices, Internet of Things (IoT) and Cloud Technologies. 
You can find more details at the link below:


https://s.apache.org/0hnG

Ticket prices will be going up on 8^th June 2018, so please make sure 
that you register soon if you want to beat the price increase. 
https://foss-backstage.de/tickets


Remember that registering for the Apache EU Roadshow also gives you 
access to FOSS Backstage so you can attend any talks and workshops from 
both conferences. And don’t forget that our Apache Lounge will be open 
throughout the whole conference as a place to meet up, hack and relax.


We look forward to seeing you in Berlin!

Thanks
Sharan Foga,  VP Apache Community Development

http://apachecon.com/
@apachecon

PLEASE NOTE: You are receiving this message because you are subscribed 
to a user@ or dev@ list of one or more Apache Software Foundation projects.

[jira] [Created] (HIVE-19755) insertsel_fail.q.out needs to be updated on branch-3

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-19755:
--

 Summary: insertsel_fail.q.out needs to be updated on branch-3
 Key: HIVE-19755
 URL: https://issues.apache.org/jira/browse/HIVE-19755
 Project: Hive
  Issue Type: Bug
Reporter: Vineet Garg
Assignee: Vineet Garg






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19754) vector_decimal_2 failing on branch-3

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-19754:
--

 Summary: vector_decimal_2 failing on branch-3
 Key: HIVE-19754
 URL: https://issues.apache.org/jira/browse/HIVE-19754
 Project: Hive
  Issue Type: Bug
Reporter: Vineet Garg
Assignee: Vineet Garg


caused by HIVE-19108. This needs golden file update only on branch-3



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19753) Strict managed tables mode in Hive

[Jason Dere (JIRA)]

Jason Dere created HIVE-19753:
-

 Summary: Strict managed tables mode in Hive
 Key: HIVE-19753
 URL: https://issues.apache.org/jira/browse/HIVE-19753
 Project: Hive
  Issue Type: Bug
Reporter: Jason Dere
Assignee: Jason Dere


Create a mode in Hive which enforces that all managed tables are transactional 
(both full or insert-only tables allowed). Non-transactional tables, as well as 
non-native tables, must be created as external tables when this mode is enabled.
The idea would be that in strict managed tables mode all of the data written to 
managed tables would have been done through Hive.
The mode would be enabled using config setting hive.strict.managed.tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19752) PerfLogger integration for critical Hive-on-S3 paths

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-19752:
---

 Summary: PerfLogger integration for critical Hive-on-S3 paths
 Key: HIVE-19752
 URL: https://issues.apache.org/jira/browse/HIVE-19752
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Reporter: Sahil Takiar
Assignee: Sahil Takiar


There are several areas where Hive performs a lot of S3 operations, it would be 
good to add PerfLogger statements around this so we can measure how long they 
take.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19751) create submodule of hive-upgrade-acid for preUpgrade and postUpgrade

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-19751:
-

 Summary: create submodule of hive-upgrade-acid for preUpgrade and 
postUpgrade
 Key: HIVE-19751
 URL: https://issues.apache.org/jira/browse/HIVE-19751
 Project: Hive
  Issue Type: Sub-task
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


Basically need to produce 2 separate jars: 1 for pre-upgrade step that can be 
compiled/unit tested with 2.x jars and another can be compiled/tested with 3.x 
jars. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19750) Initialize NEXT_WRITE_ID. NWI_NEXT on converting an existing table to full acid

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-19750:
-

 Summary: Initialize NEXT_WRITE_ID. NWI_NEXT on converting an 
existing table to full acid
 Key: HIVE-19750
 URL: https://issues.apache.org/jira/browse/HIVE-19750
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
 Fix For: 3.1.0


Need to set this to a reasonably high value the the table.
This will reserve a range of write IDs that will be treated by the system as 
committed.
This is needed so that we can assign unique ROW__IDs to each row in files that 
already exist in the table.  For example, if the value is initialized to the 
number of files currently in the table, we can think of each file as written by 
a separate transaction and thus a free to assign bucketProperty (BucketCodec) 
of ROW_ID in whichever way is convenient.
it's guaranteed that all rows get unique ROW_IDs this way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67220: HIVE-12192

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67220/
---

(Updated May 31, 2018, 7:23 p.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-12192
https://issues.apache.org/jira/browse/HIVE-12192


Repository: hive-git


Description
---

HIVE-12192


Diffs (updated)
-

  
accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/mr/TestHiveAccumuloTypes.java
 926f5720ac6ff0cfaf3858f2eba4e9e4ef37a889 
  common/src/java/org/apache/hadoop/hive/common/type/Date.java PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/type/Timestamp.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/type/TimestampTZUtil.java 
90ffddba0dd6c608857ee645023532de86c731b8 
  common/src/java/org/apache/hadoop/hive/common/type/TimestampUtils.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
3295d1dbc57101df2f4c9f035a288a0488f5059f 
  common/src/java/org/apache/hive/common/util/DateParser.java 
949fdbafcfd599e47889297955ed1e39b4f747b5 
  common/src/java/org/apache/hive/common/util/TimestampParser.java 
f674b5d30bd8addde1dad70dba3553651c0bea60 
  
common/src/test/org/apache/hadoop/hive/common/type/TestHiveDecimalOrcSerializationUtils.java
 72dce4deaaef819a436aa0ac6bdf3f526ff5aba5 
  common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java 
5a3f0481bc23e807f1f5e7b12cb9e5422b7f0e4a 
  common/src/test/org/apache/hive/common/util/TestDateParser.java 
0553b3d38718b4b4fabc808a607a98e3aa00efd4 
  common/src/test/org/apache/hive/common/util/TestTimestampParser.java 
c982af65c6228bf515aa06c2b586a7651e100ccc 
  data/files/alltypesorc3xcols e48487328bcac6872ce7cc652b5abed2529d538b 
  data/files/orc_split_elim.orc cd145d343104983b4b09603c251ee749e5f82cc7 
  druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidSerDe.java 
842a9fada6e411c4ac0e01328af11f935da3860e 
  druid-handler/src/test/org/apache/hadoop/hive/druid/serde/TestDruidSerDe.java 
e45de0f93f741b10d2f49cd7e36260f55c77edec 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java 
bc4e1466f513bc1d009f6ce64ce24e1cae592e44 
  
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseInputFormatUtil.java
 05cc30a62137733e7e94cb8d61c79503ec4f16e2 
  hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java 
d94dbe8d8aadc78cfc9b60799165f378dcc8486b 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/DataType.java 
6dcee4024ba16da30fb55a8c07a2439292324115 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/JsonSerDe.java 
114c205c83cefa23827af4072d0dc1dc4056d83c 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/ReaderWriter.java 
cb1c459afbd6a42e245728cbd782ea9261b4e85e 
  
hcatalog/hcatalog-pig-adapter/src/main/java/org/apache/hive/hcatalog/pig/HCatBaseStorer.java
 ec620d2fe06ebe14cf6ce03bbcfce877e34651a9 
  
itests/hive-jmh/src/main/java/org/apache/hive/benchmark/vectorization/ColumnVectorGenUtil.java
 d80b6d43fe44b19401a79d5190935bbabb9d522e 
  
llap-common/src/test/org/apache/hadoop/hive/llap/io/TestChunkedInputStream.java 
77559e1e589a67cad1bbc90ff190ffd5ec1db74f 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/GenericColumnVectorProducer.java
 7af1b050cec3bf0573902930078cea2528f29502 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 feccb878b79915c3442d514a58995a85a4b2dcd0 
  ql/src/gen/vectorization/ExpressionTemplates/DTIColumnCompareScalar.txt 
0d3ee2b74c9cdaeb4c07fc384a43603e543d5371 
  ql/src/gen/vectorization/ExpressionTemplates/DTIScalarCompareColumn.txt 
be5f641291e169822e8682aa0e1373d276cf50d7 
  
ql/src/gen/vectorization/ExpressionTemplates/DateColumnArithmeticIntervalYearMonthColumn.txt
 32dd6ed69f55ed2632447cdc6dedd73df9e9be33 
  
ql/src/gen/vectorization/ExpressionTemplates/DateColumnArithmeticIntervalYearMonthScalar.txt
 94c0c5c86fd3a71aa308d3c153b40db29a8bebb2 
  
ql/src/gen/vectorization/ExpressionTemplates/DateColumnArithmeticTimestampColumn.txt
 96c525d2b534da76ceff37f3ed293cad80c3fcaf 
  
ql/src/gen/vectorization/ExpressionTemplates/DateColumnArithmeticTimestampScalar.txt
 fb22992657ca9cec333487082a65ca6273a8cc47 
  
ql/src/gen/vectorization/ExpressionTemplates/DateScalarArithmeticIntervalYearMonthColumn.txt
 0c8ec9c161203513cfa84aa51151e5e187dc2739 
  
ql/src/gen/vectorization/ExpressionTemplates/DateScalarArithmeticTimestampColumn.txt
 ef8f2a3965c02989cd5bc00209bdc75a27b61b0f 
  
ql/src/gen/vectorization/ExpressionTemplates/FilterColumnBetweenDynamicValue.txt
 5242bbd24fb70438cc5c040813e4182ea1ea25e8 
  
ql/src/gen/vectorization/ExpressionTemplates/FilterTimestampColumnCompareTimestampScalar.txt
 3fa542b7426c3e8c791b892d53021db1bf9cb5fd 
  
ql/src/gen/vectorization/ExpressionTemplates/FilterTimestampScalarCompareTimestampColumn.txt
 fcda65f246f91e115ce0104e2fde8a21174f7e2a 
  

[jira] [Created] (HIVE-19749) Acid V1 to V2 upgrade

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-19749:
-

 Summary: Acid V1 to V2 upgrade
 Key: HIVE-19749
 URL: https://issues.apache.org/jira/browse/HIVE-19749
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 3.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


umbrella jira



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19748) Add appropriate null checks to DecimalColumnStatsAggregator

[Vaibhav Gumashta (JIRA)]

Vaibhav Gumashta created HIVE-19748:
---

 Summary: Add appropriate null checks to 
DecimalColumnStatsAggregator
 Key: HIVE-19748
 URL: https://issues.apache.org/jira/browse/HIVE-19748
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Vaibhav Gumashta


In some of our internal testing, we noticed that calls to 
MetaStoreUtils.decimalToDoublee(Decimal decimal)  from within 
DecimalColumnStatsAggregator end up passing null Decimal values to the method.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19747) "GRANT ALL TO USER" failed with NullPointerException

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-19747:
---

 Summary: "GRANT ALL TO USER" failed with NullPointerException
 Key: HIVE-19747
 URL: https://issues.apache.org/jira/browse/HIVE-19747
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Affects Versions: 2.1.0
Reporter: Aihua Xu


If you issue the command 'grant all to user abc', you will see the following 
NPE exception. Seems the type in hivePrivObject is not initialized.

{noformat}
FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.isOwner(SQLAuthorizationUtils.java:265)
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLAuthorizationUtils.getPrivilegesFromMetaStore(SQLAuthorizationUtils.java:212)
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.GrantPrivAuthUtils.checkRequiredPrivileges(GrantPrivAuthUtils.java:64)
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.GrantPrivAuthUtils.authorize(GrantPrivAuthUtils.java:50)
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController.grantPrivileges(SQLStdHiveAccessController.java:179)
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessControllerWrapper.grantPrivileges(SQLStdHiveAccessControllerWrapper.java:70)
at 
org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthorizerImpl.grantPrivileges(HiveAuthorizerImpl.java:48)
at 
org.apache.hadoop.hive.ql.exec.DDLTask.grantOrRevokePrivileges(DDLTask.java:1123
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 67263: HIVE-19602

[Sahil Takiar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67263/#review204126
---


Fix it, then Ship it!




minor comment otherwise LGTM


ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java
Line 238 (original), 245 (patched)


can be private


- Sahil Takiar


On May 29, 2018, 10:53 p.m., Bharathkrishna Guruvayoor Murali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67263/
> ---
> 
> (Updated May 29, 2018, 10:53 p.m.)
> 
> 
> Review request for hive, Sahil Takiar and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-19602
> https://issues.apache.org/jira/browse/HIVE-19602
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Refactor inplace progress code in Hive-on-spark progress monitor to use 
> ProgressMonitor instance
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java 
> 7afd8864075aa0d9708274eea8839c662324c732 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkProgressMonitor.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67263/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bharathkrishna Guruvayoor Murali
> 
>

Review Request 67399: HIVE-19503 Create a test that checks for dropPartitions with directSql

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67399/
---

Review request for hive, Alexander Kolbasov and Vihang Karajgaonkar.


Bugs: HIVE-19503
https://issues.apache.org/jira/browse/HIVE-19503


Repository: hive-git


Description
---

The patch contains 2 tests:

Test which checks if the JDO cache is able to handle directSql partition drops
Test which checks if the directSQL partition drop removes every connected data 
from the RDBMS tables.
To create these tests we have 2 helper methods:

Method to create the partitioned table
Method to check the number of rows in a given RDBMS table
Added a new ObjectStore.dropPartitionsInternal method which only visible for 
testing so we can make sure that the dropPartition is using directSql and does 
not fall back to JDO.

Fixed a problem where some of the tables are not created automatically by the 
tests, adding new init queries to MetaStoreDirectSql.ensureDbInit method


Diffs
-

  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 5bb1985 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 b15d89d 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
 9912213 


Diff: https://reviews.apache.org/r/67399/diff/1/


Testing
---

Run the new tests


Thanks,

Peter Vary

[jira] [Created] (HIVE-19746) Hadoop credential provider allows to read passwords

[Oleksiy Sayankin (JIRA)]

Oleksiy Sayankin created HIVE-19746:
---

 Summary: Hadoop credential provider allows to read passwords
 Key: HIVE-19746
 URL: https://issues.apache.org/jira/browse/HIVE-19746
 Project: Hive
  Issue Type: Bug
Reporter: Oleksiy Sayankin
Assignee: Oleksiy Sayankin


This simple program allows to read any password from any {{jceks}} file:

{code}
package com.test.app;

import java.util.List;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.security.alias.CredentialProvider;
import org.apache.hadoop.security.alias.CredentialProviderFactory;

import java.io.IOException;

public class PasswordReader {
  public static void main(String[] args) throws IOException {
if (args == null || args.length == 0){
  throw new IllegalArgumentException("Credential provider path is to set");
}
String credentialProviderPath = args[0];
Configuration configuration = new Configuration();
configuration.set(CredentialProviderFactory.CREDENTIAL_PROVIDER_PATH, 
credentialProviderPath);
CredentialProvider credentialProvider = 
CredentialProviderFactory.getProviders(configuration).get(0);
List aliases = credentialProvider.getAliases();
for(String alias : aliases){
  System.out.println(alias + " = " + new 
String(configuration.getPassword(alias)));
}
  }
}
{code}

{code}
java -cp $(hadoop classpath):password-reader.jar com.test.app.PasswordReader  
jceks://hdfs/user/hive/hivemetastore.jceks
{code}

*RESULT*

{code}
javax.jdo.option.connectionpassword = 123456
{code}

File {{jceks://hdfs/user/hive/hivemetastore.jceks}} has {{\-rw\-r\-\-r\-\-}} 
permissions and {{hdfs:hdfs}} owner:group. We can't remove world readable 
permissions here, because Hive is configured for impersonation to allow users 
user than {{hdfs}} connect to HiveServer2. When removed world readable 
permissions I got the exception:

{code}
2018-05-31T10:08:40,191 ERROR [pool-7-thread-41] fs.Inode: Marking failure for: 
/user/hive/hivemetastore.jceks, error: Input/output error
2018-05-31T10:08:40,192 ERROR [pool-7-thread-41] fs.Inode: Throwing exception 
for: /user/hive/hivemetastore.jceks, error: Input/output error
2018-05-31T10:08:40,192 ERROR [pool-7-thread-41] metastore.RetryingHMSHandler: 
java.lang.RuntimeException: Error getting metastore password: null
at 
org.apache.hadoop.hive.metastore.ObjectStore.getDataSourceProps(ObjectStore.java:485)
at 
org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:279)
{code}

Any ideas how to protect passwords?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19745) Hive-site.xml is unreachable when file is not in classpath and HIVE_CONF_DIR is set incorrectly.

[Mykhailo Kysliuk (JIRA)]

Mykhailo Kysliuk created HIVE-19745:
---

 Summary: Hive-site.xml is unreachable when file is not in 
classpath and HIVE_CONF_DIR is set incorrectly.
 Key: HIVE-19745
 URL: https://issues.apache.org/jira/browse/HIVE-19745
 Project: Hive
  Issue Type: Bug
Reporter: Mykhailo Kysliuk
Assignee: Mykhailo Kysliuk
 Fix For: 2.3.3, 3.0.0, 2.1.1


If HIVE_CONF_DIR variable is not set and not in classpath, configurations have 
to rely on HIVE_HOME variable. But if HIVE_CONF_DIR is not set and HIVE_HOME is 
set, configs could not find hive-site.xml file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-19744) In Beeline if -u is specified the default connection should not be tried at all

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-19744:
---

 Summary: In Beeline if -u is specified the default connection 
should not be tried at all
 Key: HIVE-19744
 URL: https://issues.apache.org/jira/browse/HIVE-19744
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Reporter: Zoltan Haindrich


I wanted to explicitly connect to a hiveserver by specifying {{-u}} but that 
didn't work because it was not running/etc...
The strange thing is that somehow the default connection is activated...and 
tried

The possible "hazard" here is that if someone specifies {{-u $MY_DEV_HS2 -f 
recreate_db.sql}} to run some sql script...beeline may connect somewhere else 
and run the commands there - which might have serious consequences (in the 
above case having default as production might be interesting)

{code}
beeline -u jdbc:hive2://localhost:10502/;transportMode=binary  -n hrt_qa
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/usr/hdp/3.0.0.0-1406/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/hdp/3.0.0.0-1406/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://localhost:10502/
18/05/31 07:51:20 [main]: WARN jdbc.HiveConnection: Failed to connect to 
localhost:10502
Unknown HS2 problem when communicating with Thrift server.
Error: Could not open client transport with JDBC Uri: 
jdbc:hive2://localhost:10502/: Invalid status 72 (state=08S01,code=0)
Connecting to 
jdbc:hive2://ctr-e138-1518143905142-336795-01-16.hwx.site:2181,ctr-e138-1518143905142-336795-01-08.hwx.site:2181,ctr-e138-1518143905142-336795-01-14.hwx.site:2181,ctr-e138-1518143905142-336795-01-09.hwx.site:2181,ctr-e138-1518143905142-336795-01-15.hwx.site:2181/default;httpPath=cliservice;principal=hive/_h...@example.com;serviceDiscoveryMode=zooKeeper;ssl=true;transportMode=http;zooKeeperNamespace=hiveserver2
18/05/31 07:51:21 [main]: INFO jdbc.HiveConnection: Connected to 
ctr-e138-1518143905142-336795-01-03.hwx.site:10001
18/05/31 07:51:21 [main]: ERROR jdbc.HiveConnection: Error opening session
org.apache.thrift.transport.TTransportException: 
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: 
PKIX path validation failed: java.security.cert.CertPathValidatorException: 
signature check failed
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)