Re: Review Request 62487: HIVE-17563 CodahaleMetrics.JsonFileReporter is not updating hive.service.metrics.file.location

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62487/
---

(Updated Sept. 22, 2017, 7:25 a.m.)


Review request for hive, Aihua Xu, Sergio Pena, Sahil Takiar, and Vihang 
Karajgaonkar.


Changes
---

Fixed JsonReporter and its test


Bugs: HIVE-17563
https://issues.apache.org/jira/browse/HIVE-17563


Repository: hive-git


Description
---

HIVE-17563 CodahaleMetrics.JsonFileReporter is not updating 
hive.service.metrics.file.location

A few other changes included:

- Test clean up
- provide useful name for JSON reporter thread


Diffs (updated)
-

  
common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
 c07517a634 
  
common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestCodahaleMetrics.java
 67f81d6c43 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
 b804cdade0 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/metrics/TestMetrics.java
 259a4db439 


Diff: https://reviews.apache.org/r/62487/diff/2/

Changes: https://reviews.apache.org/r/62487/diff/1-2/


Testing
---

TestCodahaleMetrics tests that the json reporter functionality


Thanks,

Alexander Kolbasov

HIVE-9800 how to trigger

[Anishek Agarwal]

Hello,

HIVE-9800 seems to include some changes to allow testing of schema changes.
It looks like it works only on ubuntu based systems. Do anyone of you know
if its running as part of jenkins internally or even in apache how do we
trigger it to run ?


Thanks
anishek

[jira] [Created] (HIVE-17578) Create a TableRef object for Table/Partition

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-17578:
---

 Summary: Create a TableRef object for Table/Partition
 Key: HIVE-17578
 URL: https://issues.apache.org/jira/browse/HIVE-17578
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich


a quick `git grep DbName |grep -i TableName` uncovers quite a lot of places 
where the fully qualified {{schema.tablename}} is being produced

and most of the time the Table object is also present which might as well can 
have a method to service a tableref.

There might be some hidden bugs because of this...because at some places the 
fully qualified table name is produced earlier...

example callsite:
https://github.com/apache/hive/blob/266c50554b86462d8b5ac84d28a2b237b8dbfa7e/ql/src/java/org/apache/hadoop/hive/ql/hooks/UpdateInputAccessTimeHook.java#L63

and called method:
https://github.com/apache/hive/blob/266c50554b86462d8b5ac84d28a2b237b8dbfa7e/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L620





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17579) repl load without providing the database name in the command fails.

[anishek (JIRA)]

anishek created HIVE-17579:
--

 Summary: repl load without providing the database name in the 
command fails.
 Key: HIVE-17579
 URL: https://issues.apache.org/jira/browse/HIVE-17579
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 3.0.0
Reporter: anishek
 Fix For: 3.0.0


repl dump [databasename] = > [hdfs location]

if we run {{repl load [hdfs location]}} this fails. it should pick the database 
name from the metadata file in this case but leads to a HiveException



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 62453: HIVE-17371 : Move tokenstores to metastore module

[Vihang Karajgaonkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62453/
---

Review request for hive, Aihua Xu, Alan Gates, and Thejas Nair.


Bugs: HIVE-17371
https://issues.apache.org/jira/browse/HIVE-17371


Repository: hive-git


Description
---

HIVE-17371 : Move tokenstores to metastore module

I think it makes more sense to move the tokenstore implementations to metastore 
instead of having two copies of each in Hive and metastore. The TokenManager 
and SecretManager classes can be kept as is to each consuming applications like 
HS2 or Metastore itself so that future modifications can be done independently.
Here is the approach I followed.
1. Moved the HiveDelegationTokenManager and 
TokenStoreDelegationTokenSecretManager from shims-common to service module 
since Metastore anyways has its own copy of TokenManager and Metastore cannot 
have a dependency on shims-common.
2. Moved the MemoryTokenStore, ZooKeeperTokenStore and DBTokenStore to 
standalone-metastore module. In order for these tokenstores to be consumable 
for multiple applications like HS2 and Metastore (and hopefully other 
applications in the future) we need to operate of higher level abstraction of 
AbstractDelegationTokenIdentifier and DelegationTokenInformation.
3. Each application can define its DelegationTokenIndentifier like Hive already 
does in DelegationTokenIdentifier (may be we should rename this to 
HiveDelegationTokenIdentifier to make it more explicit. Similarly, metastore 
has its own implementation of DelegationTokenIdentifier in 
DelegationTokenIdentifier (again we should rename it possibly to say 
MetastoreDelegationTokenIndentifier)
4. Modified the shims-common classes of DelegationTokenSecretManager and 
DelegationTokenStore to operate on hadoop level abstractions instead of Hive 
defined ones.


Diffs
-

  
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestHiveAuthFactory.java
 e3a51909df14d90a538426163f332a6378f35f40 
  
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithDBTokenStore.java
 d690aaa673a50785561750f4f461ec867b6f0abc 
  
itests/hive-unit-hadoop2/src/test/java/org/apache/hadoop/hive/metastore/security/TestHadoopAuthBridge23.java
 ef360400a1873ab142cb9cbdce045cbed394af0e 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/thrift/TestDBTokenStore.java
 4bfa22419cb1ea5abe56773c1c76186b01996e03 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java
 7800416a60239246e96497229d592736f2a9d79d 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
cf33cca24ffc7583f2be62bb79815b5e43b2ff42 
  service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java 
541fe5ec8da9ba66ac44c65e6fe0dcd3c601943f 
  shims/common/src/main/java/org/apache/hadoop/hive/shims/Utils.java 
3c93186082ae363497dc94ca1303db4f746fd8b5 
  shims/common/src/main/java/org/apache/hadoop/hive/thrift/DBTokenStore.java 
d6dc0796e77591d3afca8dbd29c3aa0eff255dd0 
  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java
 5299e18743aa45c539287b335f95e8ce8df0fc35 
  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenStore.java
 867b4ed98859f681770fccfe7478ab519c823924 
  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java
 6c8b362270e48cb5641d77b09f1a19d875df0cbb 
  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/HiveDelegationTokenManager.java
 b3e4a7608282be603e79d1d101679e239a5219b0 
  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/MemoryTokenStore.java 
9d837b8fa19ed53546c4a95944484cc1f06a21e6 
  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/TokenStoreDelegationTokenSecretManager.java
 4719b85e8d2f9045ae61e0e86c6624b865aa2653 
  
shims/common/src/main/java/org/apache/hadoop/hive/thrift/ZooKeeperTokenStore.java
 885ec56d543e1c46e74dc148c73f249803b7604c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenStore.java
 0cafeff89614c94ba551a5f1ba9c4d892ee5720a 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/MemoryTokenStore.java
 c484cd3132d08c8c0493fa98e79fa449e84ef9b8 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/MetastoreDelegationTokenManager.java
 2b0110fe0badcb571144067539031fb6ac81276b 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java
 4abcec789476982389b3c0bc34d4990c59dbf9da 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/SecurityUtils.java
 9f0ca829b1e3a469a3275c54a74d90278fc030fa 


Diff: https://reviews.apache.org/r/62453/diff/1/


Testing
---

HiveQA


Thanks,

Vihang Karajgaonkar

Re: Review Request 62442: HIVE-17569: Compare filtered output files in BeeLine tests

[Peter Vary]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62442/#review185995
---



Overall looks good. Just a few nits, which might be a matter of taste anyway. 
Feel free to object, it you find it unreasonable.

Thanks for the patch!


itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java
Lines 91 (patched)


I think it would be nice to have the default value as a boolean instead of 
a string, and we might want to call this method getBooleanPropertyValue. What 
do you think?



itests/util/src/main/java/org/apache/hive/beeline/QFile.java
Lines 70 (patched)


We might want to use regexps here where we have separators like 
DESCRIBE[\s\n]+EXTENDED - just an example, which probably should be changed to 
be valid :)



itests/util/src/main/java/org/apache/hive/beeline/QFile.java
Lines 328 (patched)


nit: Could we use a proper javadoc comment? Thanks!



itests/util/src/main/java/org/apache/hive/beeline/QFile.java
Lines 416 (patched)


nit: Javadoc comment please :)


- Peter Vary


On Sept. 20, 2017, 3:11 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62442/
> ---
> 
> (Updated Sept. 20, 2017, 3:11 p.m.)
> 
> 
> Review request for hive and Peter Vary.
> 
> 
> Bugs: HIVE-17569
> https://issues.apache.org/jira/browse/HIVE-17569
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Introduce a new property "test.beeline.compare.portable" with the default 
> value false and if this property is set to true, the result of the commands 
> "EXPLAIN", "DESCRIBE EXTENDED" and "DESCRIBE FORMATTED" will be filtered out 
> from the out files before comparing them in BeeLine tests.
> 
> 
> Diffs
> -
> 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java
>  9dfc253 
>   itests/util/src/main/java/org/apache/hive/beeline/QFile.java e70ac38 
> 
> 
> Diff: https://reviews.apache.org/r/62442/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: TestMiniAccumuloCliDriver hangs

[Peter Vary]

Hi,

Last time I discussed this with Ashutosh I mentioned that Cloudera supports 
Accumulo, so it would be good if we can have it here too.
I remember not too long ago some commit related to enhancing index tables in 
Accumulo too.

Just using jira (when the "TestAccumuloCliDriver did not produce" starts to 
appear), I would blame this commit:
https://issues.apache.org/jira/browse/HIVE-17373 


After reverting it I was able to run the test.

Thanks,
Peter

> On Sep 20, 2017, at 4:21 AM, Sergey Shelukhin  wrote:
> 
> Hi.
> Seems like something has broken MiniAccumulo tests, making them time out;
> that, in turn, makes the whole HiveQA run slower.
> Anyone familiar with Accumulo stuff? I’ve heard rumors that it might get
> removed as a first class part of Hive in 3.0, should we just disable the
> test?
> 
> I cannot find any recent run where it has succeeded so I’m not sure what
> breaks it.
> 
> The test hangs forever and spams an exception into the logs:
> org.apache.thrift.transport.TTransportException: Failed to connect to a
> server
>at 
> org.apache.accumulo.core.client.impl.ThriftTransportPool.getAnyTransport(Th
> riftTransportPool.java:474) ~[accumulo-core-1.8.1.jar:1.8.1]
>at 
> org.apache.accumulo.core.client.impl.ServerClient.getConnection(ServerClien
> t.java:143) ~[accumulo-core-1.8.1.jar:1.8.1]
>at 
> org.apache.accumulo.core.client.impl.ServerClient.getConnection(ServerClien
> t.java:119) ~[accumulo-core-1.8.1.jar:1.8.1]
>at 
> org.apache.accumulo.core.client.impl.ServerClient.getConnection(ServerClien
> t.java:115) ~[accumulo-core-1.8.1.jar:1.8.1]
>at 
> org.apache.accumulo.core.client.impl.ServerClient.executeRaw(ServerClient.j
> ava:97) [accumulo-core-1.8.1.jar:1.8.1]
>at 
> org.apache.accumulo.core.client.impl.ServerClient.execute(ServerClient.java
> :63) [accumulo-core-1.8.1.jar:1.8.1]
>at 
> org.apache.accumulo.core.client.impl.ConnectorImpl.(ConnectorImpl.jav
> a:69) [accumulo-core-1.8.1.jar:1.8.1]
>at 
> org.apache.accumulo.core.client.ZooKeeperInstance.getConnector(ZooKeeperIns
> tance.java:270) [accumulo-core-1.8.1.jar:1.8.1]
>at 
> org.apache.accumulo.minicluster.impl.MiniAccumuloClusterImpl.getConnector(M
> iniAccumuloClusterImpl.java:749) [accumulo-minicluster-1.8.1.jar:1.8.1]
>at 
> org.apache.accumulo.minicluster.MiniAccumuloCluster.getConnector(MiniAccumu
> loCluster.java:116) [accumulo-minicluster-1.8.1.jar:1.8.1]
> 
> 
> 
> 
> 
> 

[jira] [Created] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-17580:
--

 Summary: Remove dependency of get_fields_with_environment_context 
API to serde
 Key: HIVE-17580
 URL: https://issues.apache.org/jira/browse/HIVE-17580
 Project: Hive
  Issue Type: Sub-task
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


{{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
class to access the fields metadata for the cases where it is stored along with 
the data files (avro tables). The problem is Deserializer classes is defined in 
hive-serde module and in order to make metastore independent of Hive we will 
have to remove this dependency (atleast we should change it to runtime 
dependency instead of compile time).

The other option is investigate if we can use SearchArgument to provide this 
functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62487: HIVE-17563: CodahaleMetrics.JsonFileReporter is not updating hive.service.metrics.file.location

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62487/
---

(Updated Sept. 22, 2017, 4:01 p.m.)


Review request for hive, Aihua Xu, Sergio Pena, Sahil Takiar, and Vihang 
Karajgaonkar.


Changes
---

removed unused imports


Summary (updated)
-

HIVE-17563: CodahaleMetrics.JsonFileReporter is not updating 
hive.service.metrics.file.location


Bugs: HIVE-17563
https://issues.apache.org/jira/browse/HIVE-17563


Repository: hive-git


Description (updated)
---

HIVE-17563: CodahaleMetrics.JsonFileReporter is not updating 
hive.service.metrics.file.location


Diffs (updated)
-

  
common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
 c07517a634e35c936d6ea68e9a2874ac2c59929a 
  
common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestCodahaleMetrics.java
 67f81d6c43a7904dc384cfcce6f948f1f802144c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
 b804cdade07079955a65cb431fab078dcecd53b1 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/metrics/TestMetrics.java
 259a4db43905c47d873776df96210a4e77d07076 


Diff: https://reviews.apache.org/r/62487/diff/3/

Changes: https://reviews.apache.org/r/62487/diff/2-3/


Testing
---

TestCodahaleMetrics tests that the json reporter functionality


Thanks,

Alexander Kolbasov

Re: Review Request 62487: HIVE-17563: CodahaleMetrics.JsonFileReporter is not updating hive.service.metrics.file.location

[Vihang Karajgaonkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62487/#review186000
---




common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
Line 53 (original), 68 (patched)


Its not clear looking at the code why we are using java.nio.file.Path 
instead of org.apache.hadoop.fs.Path. Can you please add a comment explain the 
same?


- Vihang Karajgaonkar


On Sept. 22, 2017, 4:01 p.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62487/
> ---
> 
> (Updated Sept. 22, 2017, 4:01 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Sergio Pena, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-17563
> https://issues.apache.org/jira/browse/HIVE-17563
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17563: CodahaleMetrics.JsonFileReporter is not updating 
> hive.service.metrics.file.location
> 
> 
> Diffs
> -
> 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
>  c07517a634e35c936d6ea68e9a2874ac2c59929a 
>   
> common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestCodahaleMetrics.java
>  67f81d6c43a7904dc384cfcce6f948f1f802144c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
>  b804cdade07079955a65cb431fab078dcecd53b1 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/metrics/TestMetrics.java
>  259a4db43905c47d873776df96210a4e77d07076 
> 
> 
> Diff: https://reviews.apache.org/r/62487/diff/3/
> 
> 
> Testing
> ---
> 
> TestCodahaleMetrics tests that the json reporter functionality
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: HIVE-9800 how to trigger

[Vihang Karajgaonkar]

The jenkins job to test schema changes is currently not working. HIVE-15349
 was created to fix that.
I think we should fix that soon, I remember I had fixed some simple syntax
errors in our db scripts which would have been caught if this job was
working. One less thing to take of ..

On Thu, Sep 21, 2017 at 10:51 PM, Anishek Agarwal  wrote:

> Hello,
>
> HIVE-9800 seems to include some changes to allow testing of schema changes.
> It looks like it works only on ubuntu based systems. Do anyone of you know
> if its running as part of jenkins internally or even in apache how do we
> trigger it to run ?
>
>
> Thanks
> anishek
>

[jira] [Created] (HIVE-17581) Replace some calcite dependencies with native ones

[slim bouguerra (JIRA)]

slim bouguerra created HIVE-17581:
-

 Summary: Replace some calcite dependencies with native ones
 Key: HIVE-17581
 URL: https://issues.apache.org/jira/browse/HIVE-17581
 Project: Hive
  Issue Type: Sub-task
  Components: Druid integration
Affects Versions: 3.0.0
Reporter: slim bouguerra
Assignee: slim bouguerra


This is a followup of HIVE-17468. This patch excludes some unwanted 
druid-calcite dependencies.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17582) Followup of HIVE-15708

[slim bouguerra (JIRA)]

slim bouguerra created HIVE-17582:
-

 Summary: Followup of HIVE-15708
 Key: HIVE-17582
 URL: https://issues.apache.org/jira/browse/HIVE-17582
 Project: Hive
  Issue Type: Bug
  Components: Druid integration
Affects Versions: 3.0.0
Reporter: slim bouguerra
Assignee: slim bouguerra


HIVE-15708 commit be59e024420ed5ca970e87a6dec402fecee21f06 
introduced some unwanted bugs
it did change the following code 
org.apache.hadoop.hive.druid.io.DruidQueryBasedInputFormat#169
{code}
  builder.intervals(Arrays.asList(DruidTable.DEFAULT_INTERVAL));
{code}
with 
{code}
final List intervals = Arrays.asList();
builder.intervals(intervals);
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62487: HIVE-17563: CodahaleMetrics.JsonFileReporter is not updating hive.service.metrics.file.location

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62487/
---

(Updated Sept. 22, 2017, 6:14 p.m.)


Review request for hive, Aihua Xu, Sergio Pena, Sahil Takiar, and Vihang 
Karajgaonkar.


Changes
---

- Added comments to address code review comments from Vihang.


Bugs: HIVE-17563
https://issues.apache.org/jira/browse/HIVE-17563


Repository: hive-git


Description
---

HIVE-17563: CodahaleMetrics.JsonFileReporter is not updating 
hive.service.metrics.file.location


Diffs (updated)
-

  
common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
 c07517a634e35c936d6ea68e9a2874ac2c59929a 
  
common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestCodahaleMetrics.java
 67f81d6c43a7904dc384cfcce6f948f1f802144c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
 b804cdade07079955a65cb431fab078dcecd53b1 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/metrics/TestMetrics.java
 259a4db43905c47d873776df96210a4e77d07076 


Diff: https://reviews.apache.org/r/62487/diff/4/

Changes: https://reviews.apache.org/r/62487/diff/3-4/


Testing
---

TestCodahaleMetrics tests that the json reporter functionality


Thanks,

Alexander Kolbasov

[jira] [Created] (HIVE-17583) Fix test failure TestAccumuloCliDriver caused from the accumulo version upgrade

[Aihua Xu (JIRA)]

Aihua Xu created HIVE-17583:
---

 Summary: Fix test failure TestAccumuloCliDriver caused from the 
accumulo version upgrade
 Key: HIVE-17583
 URL: https://issues.apache.org/jira/browse/HIVE-17583
 Project: Hive
  Issue Type: Test
  Components: Test
Affects Versions: 3.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 62513: HIVE-17402 Provide more useful information in the HMS notification messages

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62513/
---

Review request for hive, Aihua Xu, Sergio Pena, Sahil Takiar, Nachiket Vaidya, 
and Vihang Karajgaonkar.


Bugs: HIVE-17402
https://issues.apache.org/jira/browse/HIVE-17402


Repository: hive-git


Description
---

HIVE-17402 Provide more useful information in the HMS notification messages


Diffs
-

  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONAddPartitionMessage.java
 5d201f465ff4b4b22d4e1838687d4a1e1c7e9cb3 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONAlterPartitionMessage.java
 7ae7d75168b0e9485992ee8a0285273a931567dd 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONAlterTableMessage.java
 f23aedf0b393d0c270596d4b9db968d6536f8a92 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONCreateDatabaseMessage.java
 6db46cf773abc02ef574ca3a7e86434fb4862d9d 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONCreateTableMessage.java
 327dc972de71c4e441f1f87af9c7b47fc4654374 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONDropDatabaseMessage.java
 bfe3f631eab9af7088cf227edc34d9c5a88443b5 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONDropPartitionMessage.java
 ecdc39d05961b2c8cb2498236340226c1f66ea32 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONDropTableMessage.java
 430062544bf118d8c996f274e14fd3435f2d3941 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONMessageFactory.java
 796273299f9eb920223fc5e0aa8806d6979982b0 


Diff: https://reviews.apache.org/r/62513/diff/1/


Testing
---


Thanks,

Alexander Kolbasov

Re: Review Request 62487: HIVE-17563: CodahaleMetrics.JsonFileReporter is not updating hive.service.metrics.file.location

[Sahil Takiar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62487/#review186011
---




common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
Line 75 (original), 116 (patched)


I think the old implementation called `shutdown()` instead of 
`shutdownNow()`. It may be best to call `shutdown()` first, call 
`awaitTermination(30s)`, and then call `shutdownNow()`.



common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
Line 77 (original), 118 (patched)


I don't know if we want this file to be deleted. The `close()` method is 
basically invoked when HS2 is shutdown, but there may be external systems who 
read the metrics after HS2 has been shutdown.



common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
Lines 157 (patched)


This looks like it should go into a `finally` block.



common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
Line 118 (original), 166 (patched)


This looks like it should go into a `finally` block.



standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
Lines 107 (patched)


Might want to change this to something more informative, like `logging 
metrics with frequency: `


- Sahil Takiar


On Sept. 22, 2017, 6:14 p.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62487/
> ---
> 
> (Updated Sept. 22, 2017, 6:14 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Sergio Pena, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-17563
> https://issues.apache.org/jira/browse/HIVE-17563
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17563: CodahaleMetrics.JsonFileReporter is not updating 
> hive.service.metrics.file.location
> 
> 
> Diffs
> -
> 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
>  c07517a634e35c936d6ea68e9a2874ac2c59929a 
>   
> common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestCodahaleMetrics.java
>  67f81d6c43a7904dc384cfcce6f948f1f802144c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
>  b804cdade07079955a65cb431fab078dcecd53b1 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/metrics/TestMetrics.java
>  259a4db43905c47d873776df96210a4e77d07076 
> 
> 
> Diff: https://reviews.apache.org/r/62487/diff/4/
> 
> 
> Testing
> ---
> 
> TestCodahaleMetrics tests that the json reporter functionality
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

[jira] [Created] (HIVE-17584) fix mapred.job.queue.name in sql standard authorization config whitelist

[Thejas M Nair (JIRA)]

Thejas M Nair created HIVE-17584:


 Summary: fix mapred.job.queue.name in sql standard authorization 
config whitelist
 Key: HIVE-17584
 URL: https://issues.apache.org/jira/browse/HIVE-17584
 Project: Hive
  Issue Type: Bug
Reporter: Thejas M Nair
Assignee: Thejas M Nair


The SQL std authorization config white list has mapred.job.queuename, it should 
be mapred.job.queue.name (see 
https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/DeprecatedProperties.html)




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17585) Improve thread safety when loading dynamic partitions in parallel

[Tao Li (JIRA)]

Tao Li created HIVE-17585:
-

 Summary: Improve thread safety when loading dynamic partitions in 
parallel
 Key: HIVE-17585
 URL: https://issues.apache.org/jira/browse/HIVE-17585
 Project: Hive
  Issue Type: Bug
Reporter: Tao Li
Assignee: Tao Li






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17586) Make HS2 BackgroundOperationPool not fixed

[Xuefu Zhang (JIRA)]

Xuefu Zhang created HIVE-17586:
--

 Summary: Make HS2 BackgroundOperationPool not fixed
 Key: HIVE-17586
 URL: https://issues.apache.org/jira/browse/HIVE-17586
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 1.1.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang


Currently the threadpool for background asynchronous operatons has a fixed size 
controled by {{hive.server2.async.exec.threads}}. However, the thread factory 
supplied for this threadpool is {{ThreadFactoryWithGarbageCleanup}} which 
creates ThreadWithGarbageCleanup. Since this is a fixed threadpool, the thread 
is actually never killed, defecting the purpose of garbage cleanup as noted in 
the thread class name. On the other hand, since these threads never go away, 
significant resources such as threadlocal variables (classloaders, hiveconfs, 
etc) are holding up even if there is no operation running. This can lead to 
escalated HS2 memory usage.

Ideally, the threadpool should not be fixed, allowing thread to die out so 
resources can be reclaimed. The existing config 
{{hive.server2.async.exec.threads}} is treated as the max, and we can add a min 
for the threadpool {{hive.server2.async.exec.min.threads}}. Default value for 
this configure is -1, which keeps the existing behavior.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17587) Remove unnecessary filter from getPartitionsFromPartitionIds call

[Tao Li (JIRA)]

Tao Li created HIVE-17587:
-

 Summary: Remove unnecessary filter from 
getPartitionsFromPartitionIds call
 Key: HIVE-17587
 URL: https://issues.apache.org/jira/browse/HIVE-17587
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Tao Li
Assignee: Tao Li
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62228: HIVE-17495: CachedStore: prewarm improvements, refactoring and caching some aggregate stats

[Vaibhav Gumashta]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62228/
---

(Updated Sept. 22, 2017, 10:30 p.m.)


Review request for hive, Ashutosh Chauhan and Thejas Nair.


Bugs: HIVE-17495
https://issues.apache.org/jira/browse/HIVE-17495


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-17495


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1ba5968 
  data/conf/hive-site.xml a205b8c 
  data/conf/llap/hive-site.xml 870b584 
  data/conf/perf-reg/hive-site.xml 012369f 
  data/conf/tez/hive-site.xml 35e8c99 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 d94d920 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
 98dad7a 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
c1a8efe 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
b51446d 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java b5e4bf0 
  metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 2bc4d99 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java 
280655d 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
7939bfe 
  metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 
80b17e0 
  
metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/BinaryColumnStatsAggregator.java
 e6c836b 
  
metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/BooleanColumnStatsAggregator.java
 a34bc9f 
  
metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/ColumnStatsAggregator.java
 a52e5e5 
  
metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
 ee95396 
  
metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DecimalColumnStatsAggregator.java
 284c12c 
  
metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DoubleColumnStatsAggregator.java
 bb4a725 
  
metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/LongColumnStatsAggregator.java
 5b1145e 
  
metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/StringColumnStatsAggregator.java
 1b29f92 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 a75dbb0 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 bbb4bf1 


Diff: https://reviews.apache.org/r/62228/diff/3/

Changes: https://reviews.apache.org/r/62228/diff/2-3/


Testing
---


Thanks,

Vaibhav Gumashta

[jira] [Created] (HIVE-17588) LlapRowRecordReader doing name-based field lookup for every column of every row

[Jason Dere (JIRA)]

Jason Dere created HIVE-17588:
-

 Summary: LlapRowRecordReader doing name-based field lookup for 
every column of every row
 Key: HIVE-17588
 URL: https://issues.apache.org/jira/browse/HIVE-17588
 Project: Hive
  Issue Type: Bug
Reporter: Jason Dere
Assignee: Jason Dere


setRowFromStruct() is using StructObjectInspector.getStructFieldRef(fieldName), 
which does a name-based lookup - this can be changed to do an index-based 
lookup which should be faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62487: HIVE-17563: CodahaleMetrics.JsonFileReporter is not updating hive.service.metrics.file.location

[Alexander Kolbasov]

> On Sept. 22, 2017, 7:48 p.m., Sahil Takiar wrote:
> > common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
> > Line 75 (original), 116 (patched)
> > 
> >
> > I think the old implementation called `shutdown()` instead of 
> > `shutdownNow()`. It may be best to call `shutdown()` first, call 
> > `awaitTermination(30s)`, and then call `shutdownNow()`.

Reverted to the old code (calling shutdown())


> On Sept. 22, 2017, 7:48 p.m., Sahil Takiar wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
> > Lines 107 (patched)
> > 
> >
> > Might want to change this to something more informative, like `logging 
> > metrics with frequency: `

Removed.


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62487/#review186011
---


On Sept. 22, 2017, 6:14 p.m., Alexander Kolbasov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62487/
> ---
> 
> (Updated Sept. 22, 2017, 6:14 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Sergio Pena, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Bugs: HIVE-17563
> https://issues.apache.org/jira/browse/HIVE-17563
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-17563: CodahaleMetrics.JsonFileReporter is not updating 
> hive.service.metrics.file.location
> 
> 
> Diffs
> -
> 
>   
> common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
>  c07517a634e35c936d6ea68e9a2874ac2c59929a 
>   
> common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestCodahaleMetrics.java
>  67f81d6c43a7904dc384cfcce6f948f1f802144c 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
>  b804cdade07079955a65cb431fab078dcecd53b1 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/metrics/TestMetrics.java
>  259a4db43905c47d873776df96210a4e77d07076 
> 
> 
> Diff: https://reviews.apache.org/r/62487/diff/4/
> 
> 
> Testing
> ---
> 
> TestCodahaleMetrics tests that the json reporter functionality
> 
> 
> Thanks,
> 
> Alexander Kolbasov
> 
>

Re: Review Request 62487: HIVE-17563: CodahaleMetrics.JsonFileReporter is not updating hive.service.metrics.file.location

[Alexander Kolbasov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62487/
---

(Updated Sept. 22, 2017, 11:46 p.m.)


Review request for hive, Aihua Xu, Carl Steinbach, Alan Gates, Sergio Pena, 
Sahil Takiar, and Vihang Karajgaonkar.


Changes
---

Addressed comments from Sahil


Bugs: HIVE-17563
https://issues.apache.org/jira/browse/HIVE-17563


Repository: hive-git


Description
---

HIVE-17563: CodahaleMetrics.JsonFileReporter is not updating 
hive.service.metrics.file.location


Diffs (updated)
-

  
common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/JsonFileMetricsReporter.java
 c07517a634e35c936d6ea68e9a2874ac2c59929a 
  
common/src/test/org/apache/hadoop/hive/common/metrics/metrics2/TestCodahaleMetrics.java
 67f81d6c43a7904dc384cfcce6f948f1f802144c 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/metrics/JsonReporter.java
 b804cdade07079955a65cb431fab078dcecd53b1 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/metrics/TestMetrics.java
 259a4db43905c47d873776df96210a4e77d07076 


Diff: https://reviews.apache.org/r/62487/diff/5/

Changes: https://reviews.apache.org/r/62487/diff/4-5/


Testing
---

TestCodahaleMetrics tests that the json reporter functionality


Thanks,

Alexander Kolbasov

[jira] [Created] (HIVE-17589) Beeline command line interface won't execute unterminated statements

[Zach Amsden (JIRA)]

Zach Amsden created HIVE-17589:
--

 Summary: Beeline command line interface won't execute unterminated 
statements
 Key: HIVE-17589
 URL: https://issues.apache.org/jira/browse/HIVE-17589
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Reporter: Zach Amsden


Beeline used to execute unterminated statements passed by command line, for 
example:

"DROP TABLE IF EXISTS functional_hbase.internal_hbase_table"

Now these statements do not execute unless terminated by a semicolon, as in:

"DROP TABLE IF EXISTS functional_hbase.internal_hbase_table;"

This is a behavioral regression that is almost certain to impact existing 
scripts, import jobs, etc.  Downstream this broke Impala's data load when using 
an updated version of Hive.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)