[jira] [Created] (HIVE-20805) Hive does not copy source data when importing as non-hive user

2018-10-24 Thread mahesh kumar behera (JIRA) 
mahesh kumar behera created HIVE-20805:
--

 Summary: Hive does not copy source data when importing as non-hive 
user 
 Key: HIVE-20805
 URL: https://issues.apache.org/jira/browse/HIVE-20805
 Project: Hive
  Issue Type: Bug
  Components: Hive, HiveServer2
Affects Versions: 4.0.0
Reporter: mahesh kumar behera
Assignee: mahesh kumar behera
 Fix For: 4.0.0


while loading data to a managed table from user given path, Hive uses move 
operation to copy data from user location to table location. In case move can 
not be used due to permission issue or mismatched encryption zone etc, hive 
uses copy and then deletes the files from source location to keep to behavior 
same. But in case the user does not have write access to the source location, 
delete will fail with file permission exception and load operation will fail. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20804) Further improvements to group by optimization with constraints

2018-10-24 Thread Vineet Garg (JIRA) 
Vineet Garg created HIVE-20804:
--

 Summary: Further improvements to group by optimization with 
constraints
 Key: HIVE-20804
 URL: https://issues.apache.org/jira/browse/HIVE-20804
 Project: Hive
  Issue Type: Improvement
  Components: Query Planning
Reporter: Vineet Garg
Assignee: Vineet Garg






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20803) Hive can't read S3 parquet file with timestamp partition

2018-10-24 Thread t oo (JIRA) 
t oo created HIVE-20803:
---

 Summary: Hive can't read S3 parquet file with timestamp partition
 Key: HIVE-20803
 URL: https://issues.apache.org/jira/browse/HIVE-20803
 Project: Hive
  Issue Type: Bug
  Components: SQL
Affects Versions: 2.3.2
Reporter: t oo


SparkThriftServer can select * from the table fine and get data. But 
HiveServer2 throws below error on select *:

 

hive.msck.path.validation = ignore in hive-site.xml
then ran msck repair my_sch.h_l

aws s3 ls s3://priv1/priv2/H_L/ --recursive
2018-10-18 03:00:56 2474983 
priv1/priv2/H_L/part_dt=20180309/part_src=xyz/part_src_file=MY_LOC/part_ldts=2018-10-18
 02:59:46/part-0-2536ca01-243c-4220-8e55-6869a045fba2.snappy.parquet


show create table my_sch.h_l;
++
| createtab_stmt |
++
| CREATE EXTERNAL TABLE `my_sch.h_l`( |
| `xy_hkey_h_l` binary, |
| `xy_rtts` timestamp, |
| `xy_rsrc` string, |
| `xy_bkcc` string, |
| `xy_mltid` string, |
| `location_id` bigint) |
| PARTITIONED BY ( |
| `part_dt` string, |
| `part_src` string, |
| `part_src_file` string, |
| `part_ldts` timestamp) |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' |
| LOCATION |
| 's3a://priv1/priv2/H_L' |
| TBLPROPERTIES ( |
| 'spark.sql.partitionProvider'='catalog', |
| 'spark.sql.sources.schema.numPartCols'='4', |
| 'spark.sql.sources.schema.numParts'='1', |
| 
'spark.sql.sources.schema.part.0'='\{"type":"struct","fields":[{"name":"xy_hkey_h_l","type":"binary","nullable":true,"metadata":{}},\{"name":"xy_rtts","type":"timestamp","nullable":true,"metadata":{}},\{"name":"xy_rsrc","type":"string","nullable":true,"metadata":{}},\{"name":"xy_bkcc","type":"string","nullable":true,"metadata":{}},\{"name":"xy_mltid","type":"string","nullable":true,"metadata":{}},\{"name":"location_id","type":"long","nullable":true,"metadata":{}},\{"name":"part_dt","type":"string","nullable":true,"metadata":{}},\{"name":"part_src","type":"string","nullable":true,"metadata":{}},\{"name":"part_src_file","type":"string","nullable":true,"metadata":{}},\{"name":"part_ldts","type":"timestamp","nullable":true,"metadata":{}}]}',
 |
| 'spark.sql.sources.schema.partCol.0'='part_dt', |
| 'spark.sql.sources.schema.partCol.1'='part_src', |
| 'spark.sql.sources.schema.partCol.2'='part_src_file', |
| 'spark.sql.sources.schema.partCol.3'='part_ldts', |
| 'transient_lastDdlTime'='1540421484') |
++


 select * from my_sch.h_l limit 5;
Error: java.io.IOException: java.lang.IllegalArgumentException: 
java.net.URISyntaxException: Relative path in absolute URI: 
part_ldts=2018-10-18 02:59:46 (state=,code=0)
org.apache.hive.service.cli.HiveSQLException: java.io.IOException: 
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path 
in absolute URI: part_ldts=2018-10-18 02:59:46
 at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
 at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)
 at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:374)
 at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:53)
 at 
org.apache.hive.beeline.IncrementalRowsWithNormalization.(IncrementalRowsWithNormalization.java:50)
 at org.apache.hive.beeline.BeeLine.print(BeeLine.java:2192)
 at org.apache.hive.beeline.Commands.executeInternal(Commands.java:1009)
 at org.apache.hive.beeline.Commands.execute(Commands.java:1205)
 at org.apache.hive.beeline.Commands.sql(Commands.java:1134)
 at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1314)
 at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1178)
 at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1033)
 at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
 at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
 at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: 
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path 
in absolute URI: part_ldts=2018-10-18 02:59:46
 at 
org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:499)
 at 
org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:307)
 at

Re: Review Request 69143: CachedStore: Add more UT coverage (outside of .q files)

2018-10-24 Thread Daniel Dai 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69143/#review210011
---




standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 94 (patched)


Can you add a comment what's the difference between isCachePrewarmed?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 227 (patched)


Log full stack



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 491 (patched)


Can you log the full stack trace (maybe using 
org.apache.hadoop.util.StringUtils.stringifyException)? There's another 
instance below.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 522 (patched)


Make maxTries a constant (and reuse it below).



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
Lines 544 (patched)


maxTries should be constant.


- Daniel Dai


On Oct. 24, 2018, 5:04 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69143/
> ---
> 
> (Updated Oct. 24, 2018, 5:04 p.m.)
> 
> 
> Review request for hive, Daniel Dai and Thejas Nair.
> 
> 
> Bugs: HIVE-20613
> https://issues.apache.org/jira/browse/HIVE-20613
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-20613
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
>  944c81313a 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
>  70490f09e7 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
>  c24e7160ac 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
>  bb20d9f42a 
>   standalone-metastore/metastore-server/src/test/resources/log4j2.properties 
> 365687e1c9 
> 
> 
> Diff: https://reviews.apache.org/r/69143/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[VOTE] Apache Hive 3.1.1 Release Candidate 0

2018-10-24 Thread Daniel Dai 
Apache Hive 3.1.1 Release Candidate 0 is available here:

http://people.apache.org/~daijy/apache-hive-3.1.1-rc-0/

Maven artifacts are available here:

https://repository.apache.org/content/repositories/orgapachehive-1092

Source tag for RCN is at:

https://github.com/apache/hive/tree/release-3.1.1-rc0

Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

Review Request 69148: HIVE-20793 add RP namespacing to workload management

2018-10-24 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69148/
---

Review request for hive, Jason Dere and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
ed6d3d80e3f6a9ba66f46223d97666b48c388028 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 c3e1e8e88c69d7713e16c7061ce8cf73a0d5e833 
  metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql 
a69046f961cdf0fff7989492c489bb62f2a66d72 
  metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql 
4c770206fe3dcceb8570be1c1ef078b376f5cafd 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
807f159daa98d40e667914adc6c53fb8ecabf998 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
4de038913a5c9a2c199f71702b8f70ca84d0856b 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java 
e57db935d9420508ed6091e12ca6b6cd3382db5d 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 aba63f050b5b98a2aeeb0df6ff2de5e6e06761f2 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 d1c0c4d1f60016f28cea69348b1b30ecb61bf083 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
4b7b61520a2d55635f474317053a17410f3a4bb7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 8cd46e3f44e7c4e47fbf7f2ce2b6350a5814106f 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 9c158040497cd3d2762620ce35e2b46bb6d5fffe 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java
 c3914b668fac18ead6196a4fc449e909f5af01b1 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 47ac68c667bea8f09f5301a6364c854bc18b3c0d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MWMResourcePlan.java
 ac51f2d77145b37da468ce8df2ac5c42f4d6c538 
  standalone-metastore/metastore-server/src/main/resources/package.jdo 
fef6a42038bb2aa0cba6dfda8d710fd37cb720e7 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 c889bbdf96b887b29be858e41ee854f0731cd5cd 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 aca5227a5bb6192da6c5f070c04d2941d636bad2 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 91ba134325094e413887a89e1d605efa99218288 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 f0d861b3a9bc982c1e24fa49415dcfc6c105cd68 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 3af2ebb253f82bb85976d229d4ac2225deffdbde 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 ee0f691b524a6e822ac14e09d24d3a49ae8565b1 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 33aa08015a9e17585c42d64d44b364be96e69eaf 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 bbb4a39ec4f6f616c9a1a9042a35cafd45cf9796 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 ea088d77fdaec85834b8fd3f01eacdfac58dd245 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 2a2d70ae802eb6f5b9ab7f4f9519a0af30d2c5b4 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 966979891b71f1cbfe50f56c40c35af8b304c47f 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 593d562c3498660861201f58d83c27d59d184046 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 4293579ad8b55d59f2230040f23e9a693d838ca7 


Diff: https://reviews.apache.org/r/69148/diff/1/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-20802) Statistics: Provide per-column statistics expanded for "explain formatted"

2018-10-24 Thread Gopal V (JIRA) 
Gopal V created HIVE-20802:
--

 Summary: Statistics: Provide per-column statistics expanded for 
"explain formatted"
 Key: HIVE-20802
 URL: https://issues.apache.org/jira/browse/HIVE-20802
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


Debugging the internal per-column statistics of Hive is challenging, 
particularly when the estimations are approximate (hive.stats.estimate=true).

Adding an extra columnStatistics field for all operators in "explain formatted" 
will allow tools to analyze the internal statistics transforms done by each 
operator without having to refer to debug logs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking

2018-10-24 Thread Gopal V (JIRA) 
Gopal V created HIVE-20801:
--

 Summary: ACID: Allow DbTxnManager to ignore non-ACID table locking
 Key: HIVE-20801
 URL: https://issues.apache.org/jira/browse/HIVE-20801
 Project: Hive
  Issue Type: Bug
Reporter: Gopal V


Enabling ACIDv1 on a cluster produces a central locking bottleneck for all 
table types, which is not always the intention.

The Hive locking for non-acid tables are advisory (i.e a client can write/read 
without locking), which means that the implementation does not offer strong 
consistency despite the lock manager consuming resources centrally.

Disabling this lock acquisition would improve the performance of non-ACID 
tables co-existing with a globally configured DbTxnManager implementation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20800) Use "posix" for property tarLongFileMode for maven-assembly-plugin

2018-10-24 Thread Wei Zheng (JIRA) 
Wei Zheng created HIVE-20800:


 Summary: Use "posix" for property tarLongFileMode for 
maven-assembly-plugin
 Key: HIVE-20800
 URL: https://issues.apache.org/jira/browse/HIVE-20800
 Project: Hive
  Issue Type: Improvement
  Components: Build Infrastructure
Affects Versions: 3.1.0
Reporter: Wei Zheng
Assignee: Wei Zheng
 Fix For: 4.0.0


Came across this error when building hive using "mvn clean install -DskipTests"

{code}

[INFO] Building tar: 
/Users/wei/apache/hive/standalone-metastore/target/apache-hive-standalone-metastore-4.0.0-SNAPSHOT-src.tar.gz
[INFO] 
[INFO] Reactor Summary:
[INFO]
[INFO] Hive Storage API 2.7.0-SNAPSHOT  SUCCESS [  5.656 s]
[INFO] Hive 4.0.0-SNAPSHOT  SUCCESS [  0.779 s]
[INFO] Hive Classifications ... SUCCESS [  0.908 s]
[INFO] Hive Shims Common .. SUCCESS [  3.217 s]
[INFO] Hive Shims 0.23  SUCCESS [  7.102 s]
[INFO] Hive Shims Scheduler ... SUCCESS [  2.069 s]
[INFO] Hive Shims . SUCCESS [  1.905 s]
[INFO] Hive Common  SUCCESS [  8.185 s]
[INFO] Hive Service RPC ... SUCCESS [  3.603 s]
[INFO] Hive Serde . SUCCESS [  7.438 s]
[INFO] Hive Standalone Metastore .. FAILURE [  0.576 s]
[INFO] Hive Standalone Metastore Common Code .. SKIPPED
[INFO] Hive Metastore . SKIPPED
[INFO] Hive Vector-Code-Gen Utilities . SKIPPED
[INFO] Hive Llap Common ... SKIPPED
[INFO] Hive Llap Client ... SKIPPED
[INFO] Hive Llap Tez .. SKIPPED
[INFO] Hive Spark Remote Client ... SKIPPED
[INFO] Hive Metastore Server .. SKIPPED
[INFO] Hive Query Language  SKIPPED
[INFO] Hive Llap Server ... SKIPPED
[INFO] Hive Service ... SKIPPED
[INFO] Hive Accumulo Handler .. SKIPPED
[INFO] Hive JDBC .. SKIPPED
[INFO] Hive Beeline ... SKIPPED
[INFO] Hive CLI ... SKIPPED
[INFO] Hive Contrib ... SKIPPED
[INFO] Hive Druid Handler . SKIPPED
[INFO] Hive HBase Handler . SKIPPED
[INFO] Hive JDBC Handler .. SKIPPED
[INFO] Hive HCatalog .. SKIPPED
[INFO] Hive HCatalog Core . SKIPPED
[INFO] Hive HCatalog Pig Adapter .. SKIPPED
[INFO] Hive HCatalog Server Extensions  SKIPPED
[INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
[INFO] Hive HCatalog Webhcat .. SKIPPED
[INFO] Hive HCatalog Streaming  SKIPPED
[INFO] Hive HPL/SQL ... SKIPPED
[INFO] Hive Streaming . SKIPPED
[INFO] Hive Llap External Client .. SKIPPED
[INFO] Hive Shims Aggregator .. SKIPPED
[INFO] Hive Kryo Registrator .. SKIPPED
[INFO] Hive TestUtils . SKIPPED
[INFO] Hive Kafka Storage Handler . SKIPPED
[INFO] Hive Packaging . SKIPPED
[INFO] Hive Metastore Tools ... SKIPPED
[INFO] Hive Metastore Tools common libraries .. SKIPPED
[INFO] Hive metastore benchmarks .. SKIPPED
[INFO] Hive Upgrade Acid .. SKIPPED
[INFO] Hive Pre Upgrade Acid 4.0.0-SNAPSHOT ... SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 42.026 s
[INFO] Finished at: 2018-10-24T15:34:40-07:00
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-assembly-plugin:3.1.0:single (assemble) on 
project hive-standalone-metastore: Execution assemble of goal 
org.apache.maven.plugins:maven-assembly-plugin:3.1.0:single failed: group id 
'74715970' is too big ( > 2097151 ). Use STAR or POSIX extensions

[GitHub] hive pull request #453: Fix the leak of lock during concurrent partition dro...

2018-10-24 Thread guangyy 
GitHub user guangyy opened a pull request:

https://github.com/apache/hive/pull/453

Fix the leak of lock during concurrent partition drop

We have seen a leaked lock on hive metastore DB which caused all
PARTITION insertion failed on timeout waiting for lock until the
metastore service is restarted.

A transaction dump on the DB shows there is a thread that is Sleep which
potentiall holds the the lock, like:
  trx_id: 33603171058
 trx_state: RUNNING
   trx_started: 2018-10-23 06:43:22
 trx_requested_lock_id: NULL
  trx_wait_started: NULL
trx_weight: 70298
   trx_mysql_thread_id: 275402202
 trx_query: NULL
   trx_operation_state: NULL
 trx_tables_in_use: 0
 trx_tables_locked: 0
  trx_lock_structs: 21286
 trx_lock_memory_bytes: 2881064
   trx_rows_locked: 98810
 trx_rows_modified: 49012
   trx_concurrency_tickets: 0
   trx_isolation_level: READ COMMITTED
 trx_unique_checks: 1
trx_foreign_key_checks: 1
trx_last_foreign_key_error: NULL
 trx_adaptive_hash_latched: 0
 trx_adaptive_hash_timeout: 0
  trx_is_read_only: 0
trx_autocommit_non_locking: 0
ID: 275402202
  USER: metastore_gold
  HOST: 10.37.182.82:36684
DB: metastoregold
   COMMAND: Sleep
  TIME: 1
 STATE:
  INFO: NULL
  duration: 1316

Given the HOST ip, we trace back to the hive metastore instance and found 
the following exceptions:
2018-10-23 06:43:22,805 WARN DataNucleus.Persistence: Exception thrown by 
StateManager.isLoaded
No such database row
org.datanucleus.exceptions.NucleusObjectNotFoundException: No such database 
row
at 
org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:357)
at 
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324)
at 
org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1120)
at 
org.datanucleus.state.JDOStateManager.loadSpecifiedFields(JDOStateManager.java:2916)
at 
org.datanucleus.state.JDOStateManager.isLoaded(JDOStateManager.java:3219)

The problem is that the caller expects a NULL if the partition does not 
exist, however, the convertToPart function would throw
an exception which lead to the leak.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guangyy/hive guang--fix-db-lock-leak

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/453.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #453


commit 7f6ace1146c32b7e6b8f175cc9c18489119c7613
Author: Guang Yang 
Date:   2018-10-24T22:28:09Z

Fix the leak of lock during concurrent partition drop

We have seen a leaked lock on hive metastore DB which caused all
PARTITION insertion failed on timeout waiting for lock until the
metastore service is restarted.

A transaction dump on the DB shows there is a thread that is Sleep which
potentiall holds the the lock, like:
  trx_id: 33603171058
 trx_state: RUNNING
   trx_started: 2018-10-23 06:43:22
 trx_requested_lock_id: NULL
  trx_wait_started: NULL
trx_weight: 70298
   trx_mysql_thread_id: 275402202
 trx_query: NULL
   trx_operation_state: NULL
 trx_tables_in_use: 0
 trx_tables_locked: 0
  trx_lock_structs: 21286
 trx_lock_memory_bytes: 2881064
   trx_rows_locked: 98810
 trx_rows_modified: 49012
   trx_concurrency_tickets: 0
   trx_isolation_level: READ COMMITTED
 trx_unique_checks: 1
trx_foreign_key_checks: 1
trx_last_foreign_key_error: NULL
 trx_adaptive_hash_latched: 0
 trx_adaptive_hash_timeout: 0
  trx_is_read_only: 0
trx_autocommit_non_locking: 0
ID: 275402202
  USER: metastore_gold
  HOST: 10.37.182.82:36684
DB: metastoregold
   COMMAND: Sleep
  TIME: 1
 STATE:
  INFO: NULL
  duration: 1316

Given the HOST ip, we trace back to the hive metastore instance and found 
the following exceptions:
2018-10-23 06:43:22,805 WARN

Re: Review Request 69107: HIVE-20512

2018-10-24 Thread Bharathkrishna Guruvayoor Murali via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69107/#review209986
---




ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java
Line 67 (original), 67 (patched)


Creating this as a threadPool of size 1. I guess that is fine, as we know 
only one thread will be used at any point?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java
Line 116 (original), 113 (patched)


Using shutDownNow instead of shutDown to cancel pending tasks


- Bharathkrishna Guruvayoor Murali


On Oct. 24, 2018, 8:55 p.m., Bharathkrishna Guruvayoor Murali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69107/
> ---
> 
> (Updated Oct. 24, 2018, 8:55 p.m.)
> 
> 
> Review request for hive, Antal Sinkovits, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Improve record and memory usage logging in SparkRecordHandler
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 
> 88dd12c05ade417aca4cdaece4448d31d4e1d65f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMergeFileRecordHandler.java
>  8880bb604e088755dcfb0bcb39689702fab0cb77 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java 
> cb5bd7ada2d5ad4f1f654cf80ddaf4504be5d035 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReduceRecordHandler.java
>  20e7ea0f4e8d4ff79dddeaab0406fc7350d22bd7 
> 
> 
> Diff: https://reviews.apache.org/r/69107/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bharathkrishna Guruvayoor Murali
> 
>

Re: Review Request 69107: HIVE-20512

2018-10-24 Thread Bharathkrishna Guruvayoor Murali via Review Board 


> On Oct. 24, 2018, 8:51 a.m., Antal Sinkovits wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java
> > Lines 56 (patched)
> > 
> >
> > Any reason why we use Timer?
> > 
> > From the timer java docs: 
> > 
> > "Java 5.0 introduced the java.util.concurrent package and one of the 
> > concurrency utilities therein is the ScheduledThreadPoolExecutor which is a 
> > thread pool for repeatedly executing tasks at a given rate or delay. It is 
> > effectively a more versatile replacement for the Timer/TimerTask 
> > combination, as it allows multiple service threads, accepts various time 
> > units, and doesn't require subclassing TimerTask (just implement Runnable). 
> > Configuring ScheduledThreadPoolExecutor with one thread makes it equivalent 
> > to Timer."

I used timer as there was only one thread, bbut I have changed it to 
ScheduledThreadPoolExecutor, as it looks more extensible


- Bharathkrishna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69107/#review209950
---


On Oct. 24, 2018, 8:55 p.m., Bharathkrishna Guruvayoor Murali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69107/
> ---
> 
> (Updated Oct. 24, 2018, 8:55 p.m.)
> 
> 
> Review request for hive, Antal Sinkovits, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Improve record and memory usage logging in SparkRecordHandler
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 
> 88dd12c05ade417aca4cdaece4448d31d4e1d65f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMergeFileRecordHandler.java
>  8880bb604e088755dcfb0bcb39689702fab0cb77 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java 
> cb5bd7ada2d5ad4f1f654cf80ddaf4504be5d035 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReduceRecordHandler.java
>  20e7ea0f4e8d4ff79dddeaab0406fc7350d22bd7 
> 
> 
> Diff: https://reviews.apache.org/r/69107/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bharathkrishna Guruvayoor Murali
> 
>

Re: Review Request 69107: HIVE-20512

2018-10-24 Thread Bharathkrishna Guruvayoor Murali via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69107/
---

(Updated Oct. 24, 2018, 8:55 p.m.)


Review request for hive, Antal Sinkovits, Sahil Takiar, and Vihang Karajgaonkar.


Changes
---

Using ThreadPoolExecutor
Addressed some review comments, and have added some comments which I want to be 
specifically reviewed.


Repository: hive-git


Description
---

Improve record and memory usage logging in SparkRecordHandler


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 
88dd12c05ade417aca4cdaece4448d31d4e1d65f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMergeFileRecordHandler.java
 8880bb604e088755dcfb0bcb39689702fab0cb77 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java 
cb5bd7ada2d5ad4f1f654cf80ddaf4504be5d035 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReduceRecordHandler.java 
20e7ea0f4e8d4ff79dddeaab0406fc7350d22bd7 


Diff: https://reviews.apache.org/r/69107/diff/2/

Changes: https://reviews.apache.org/r/69107/diff/1-2/


Testing
---


Thanks,

Bharathkrishna Guruvayoor Murali

Re: Review Request 69107: HIVE-20512

2018-10-24 Thread Bharathkrishna Guruvayoor Murali via Review Board 


> On Oct. 23, 2018, 7:50 p.m., Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java
> > Line 50 (original), 53 (patched)
> > 
> >
> > this need to be volatile since it is modified by the timer task

This variable is also used as 
logThresholdInterval = Math.min(maxLogThresholdInterval, 2 * 
logThresholdInterval);

Non-atomic operation. So should I make this variable atomic as well?


- Bharathkrishna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69107/#review209935
---


On Oct. 20, 2018, 7:13 p.m., Bharathkrishna Guruvayoor Murali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69107/
> ---
> 
> (Updated Oct. 20, 2018, 7:13 p.m.)
> 
> 
> Review request for hive, Antal Sinkovits, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Improve record and memory usage logging in SparkRecordHandler
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 
> 88dd12c05ade417aca4cdaece4448d31d4e1d65f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMergeFileRecordHandler.java
>  8880bb604e088755dcfb0bcb39689702fab0cb77 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java 
> cb5bd7ada2d5ad4f1f654cf80ddaf4504be5d035 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReduceRecordHandler.java
>  20e7ea0f4e8d4ff79dddeaab0406fc7350d22bd7 
> 
> 
> Diff: https://reviews.apache.org/r/69107/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bharathkrishna Guruvayoor Murali
> 
>

Re: Review Request 69107: HIVE-20512

2018-10-24 Thread Bharathkrishna Guruvayoor Murali via Review Board 


> On Oct. 23, 2018, 7:50 p.m., Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java
> > Line 49 (original), 52 (patched)
> > 
> >
> > i think volatile long is sufficient here and is probably cheaper. 
> > atomics might be expensive when done per row

I first used volatile, but I replaced it with AtomicLong because the rowNumber 
needs to be incremented and rowNumber++ on a volatile variable is not 
considered a safe operation. What do you think about that?


- Bharathkrishna


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69107/#review209935
---


On Oct. 20, 2018, 7:13 p.m., Bharathkrishna Guruvayoor Murali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69107/
> ---
> 
> (Updated Oct. 20, 2018, 7:13 p.m.)
> 
> 
> Review request for hive, Antal Sinkovits, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Improve record and memory usage logging in SparkRecordHandler
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 
> 88dd12c05ade417aca4cdaece4448d31d4e1d65f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMergeFileRecordHandler.java
>  8880bb604e088755dcfb0bcb39689702fab0cb77 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java 
> cb5bd7ada2d5ad4f1f654cf80ddaf4504be5d035 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReduceRecordHandler.java
>  20e7ea0f4e8d4ff79dddeaab0406fc7350d22bd7 
> 
> 
> Diff: https://reviews.apache.org/r/69107/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bharathkrishna Guruvayoor Murali
> 
>

Review Request 69145: OrcInputFormat.SplitGenerator.callInternal() can be optimized

2018-10-24 Thread Igor Kryvenko 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69145/
---

Review request for hive and Eugene Koifman.


Bugs: HIVE-14516
https://issues.apache.org/jira/browse/HIVE-14516


Repository: hive-git


Description
---

callIntenal() has 
// We can't eliminate stripes if there are deltas because the
// deltas may change the rows making them match the predicate.

but in Acid 2.0, the deltas only have delete events thus eliminating stripes 
from "base" of split should be safe.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 25b2d483d7 


Diff: https://reviews.apache.org/r/69145/diff/1/


Testing
---


Thanks,

Igor Kryvenko

[jira] [Created] (HIVE-20799) Fix or disable TestRpc.testClientTimeout

2018-10-24 Thread Vihang Karajgaonkar (JIRA) 
Vihang Karajgaonkar created HIVE-20799:
--

 Summary: Fix or disable TestRpc.testClientTimeout
 Key: HIVE-20799
 URL: https://issues.apache.org/jira/browse/HIVE-20799
 Project: Hive
  Issue Type: Test
Reporter: Vihang Karajgaonkar


Test failed without any code changes on master. See HIVE-20798



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Precommit build timing out

2018-10-24 Thread Jesus Camacho Rodriguez 
Thanks Vihang!

On 10/24/18, 12:32 PM, "Vihang Karajgaonkar"  
wrote:

Restarted ptest server and confirmed its back up and running. There was
some outage on the jenkins side as well as GCE yesterday. Not sure what
caused it really, but looks ok now.

On Tue, Oct 23, 2018 at 4:48 PM, Vihang Karajgaonkar 
wrote:

> I took a quick look at it from the server side. It seems ok. I think the
> jenkins node is not able to ping the ptest server.
>
> On Tue, Oct 23, 2018 at 4:13 PM, Jesus Camacho Rodriguez <
> jcamachorodrig...@hortonworks.com> wrote:
>
>> Hi,
>>
>> Last three builds have timed out after 8 hours. Any idea on what might be
>> going on?
>>
>> …
>> + local 'PTEST_CLASSPATH=/home/jenkins/jenkins-slave/workspace/
>> PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/
>> target/hive-ptest-3.0-classes.jar:/home/jenkins/jenkins-
>> slave/workspace/PreCommit-HIVE-Build/hive/build/hive/tes
>> tutils/ptest2/target/lib/*'
>> + java -cp '/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/
>> hive/build/hive/testutils/ptest2/target/hive-ptest-3.0-
>> classes.jar:/home/jenkins/jenkins-slave/workspace/
>> PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/lib/*'
>> org.apache.hive.ptest.api.client.PTestClient --command testStart
>> --outputDir /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/
>> hive/build/hive/testutils/ptest2/target --password '[***]'
>> --testHandle PreCommit-HIVE-Build-14608 --endpoint
>> http://104.198.109.242:8080/hive-ptest-3.0 --logsEndpoint
>> http://104.198.109.242/logs/ --profile master-mr2 --patch
>> https://issues.apache.org/jira/secure/attachment/12945103/
>> HIVE-20772.02.patch --jira HIVE-20772
>> Build timed out (after 500 minutes). Marking the build as aborted.
>> …
>>
>> Thanks,
>> Jesús
>>
>
>

Re: Precommit build timing out

2018-10-24 Thread Vihang Karajgaonkar 
Restarted ptest server and confirmed its back up and running. There was
some outage on the jenkins side as well as GCE yesterday. Not sure what
caused it really, but looks ok now.

On Tue, Oct 23, 2018 at 4:48 PM, Vihang Karajgaonkar 
wrote:

> I took a quick look at it from the server side. It seems ok. I think the
> jenkins node is not able to ping the ptest server.
>
> On Tue, Oct 23, 2018 at 4:13 PM, Jesus Camacho Rodriguez <
> jcamachorodrig...@hortonworks.com> wrote:
>
>> Hi,
>>
>> Last three builds have timed out after 8 hours. Any idea on what might be
>> going on?
>>
>> …
>> + local 'PTEST_CLASSPATH=/home/jenkins/jenkins-slave/workspace/
>> PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/
>> target/hive-ptest-3.0-classes.jar:/home/jenkins/jenkins-
>> slave/workspace/PreCommit-HIVE-Build/hive/build/hive/tes
>> tutils/ptest2/target/lib/*'
>> + java -cp '/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/
>> hive/build/hive/testutils/ptest2/target/hive-ptest-3.0-
>> classes.jar:/home/jenkins/jenkins-slave/workspace/
>> PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target/lib/*'
>> org.apache.hive.ptest.api.client.PTestClient --command testStart
>> --outputDir /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/
>> hive/build/hive/testutils/ptest2/target --password '[***]'
>> --testHandle PreCommit-HIVE-Build-14608 --endpoint
>> http://104.198.109.242:8080/hive-ptest-3.0 --logsEndpoint
>> http://104.198.109.242/logs/ --profile master-mr2 --patch
>> https://issues.apache.org/jira/secure/attachment/12945103/
>> HIVE-20772.02.patch --jira HIVE-20772
>> Build timed out (after 500 minutes). Marking the build as aborted.
>> …
>>
>> Thanks,
>> Jesús
>>
>
>

[jira] [Created] (HIVE-20798) Test precommit job

2018-10-24 Thread Vihang Karajgaonkar (JIRA) 
Vihang Karajgaonkar created HIVE-20798:
--

 Summary: Test precommit job
 Key: HIVE-20798
 URL: https://issues.apache.org/jira/browse/HIVE-20798
 Project: Hive
  Issue Type: Test
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


Dummy patch request to test if precommit works after restart



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Review Request 69143: CachedStore: Add more UT coverage (outside of .q files)

2018-10-24 Thread Vaibhav Gumashta 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69143/
---

Review request for hive, Daniel Dai and Thejas Nair.


Bugs: HIVE-20613
https://issues.apache.org/jira/browse/HIVE-20613


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-20613


Diffs
-

  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
 944c81313a 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
 70490f09e7 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
 c24e7160ac 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
 bb20d9f42a 
  standalone-metastore/metastore-server/src/test/resources/log4j2.properties 
365687e1c9 


Diff: https://reviews.apache.org/r/69143/diff/1/


Testing
---


Thanks,

Vaibhav Gumashta

[jira] [Created] (HIVE-20797) Print Number of Locks Acquired

2018-10-24 Thread BELUGA BEHR (JIRA) 
BELUGA BEHR created HIVE-20797:
--

 Summary: Print Number of Locks Acquired
 Key: HIVE-20797
 URL: https://issues.apache.org/jira/browse/HIVE-20797
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2, Locking
Affects Versions: 4.0.0
Reporter: BELUGA BEHR


The number of locks acquired by a query can greatly influence the performance 
and stability of the system, especially for ZK locks.  Please add INFO level 
logging with the number of locks each query obtains.

Log here:
https://github.com/apache/hive/blob/3963c729fabf90009cb67d277d40fe5913936358/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L1670-L1672

{quote}
A list of acquired locks will be stored in the 
org.apache.hadoop.hive.ql.Context object and can be retrieved via 
org.apache.hadoop.hive.ql.Context#getHiveLocks.
{quote}

https://github.com/apache/hive/blob/758ff449099065a84c46d63f9418201c8a6731b1/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManager.java#L115-L127



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20796) jdbc URL can contain sensitive information that should not be logged

2018-10-24 Thread Laszlo Pinter (JIRA) 
Laszlo Pinter created HIVE-20796:


 Summary: jdbc URL can contain sensitive information that should 
not be logged
 Key: HIVE-20796
 URL: https://issues.apache.org/jira/browse/HIVE-20796
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Affects Versions: 4.0.0
Reporter: Laszlo Pinter
Assignee: Laszlo Pinter


It is possible to put passwords in the jdbc connection url and some jdbc 
drivers will supposedly use that. (derby, mysql). This information is 
considered sensitive, and should be masked out, while logging the connection 
url.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20795) Rename CompileLockTest.java to follow the test's naming convention.

2018-10-24 Thread Denys Kuzmenko (JIRA) 
Denys Kuzmenko created HIVE-20795:
-

 Summary: Rename CompileLockTest.java to follow the test's naming 
convention.
 Key: HIVE-20795
 URL: https://issues.apache.org/jira/browse/HIVE-20795
 Project: Hive
  Issue Type: Bug
Reporter: Denys Kuzmenko






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69107: HIVE-20512

2018-10-24 Thread Antal Sinkovits via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69107/#review209950
---




ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java
Lines 56 (patched)


Any reason why we use Timer?

From the timer java docs: 

"Java 5.0 introduced the java.util.concurrent package and one of the 
concurrency utilities therein is the ScheduledThreadPoolExecutor which is a 
thread pool for repeatedly executing tasks at a given rate or delay. It is 
effectively a more versatile replacement for the Timer/TimerTask combination, 
as it allows multiple service threads, accepts various time units, and doesn't 
require subclassing TimerTask (just implement Runnable). Configuring 
ScheduledThreadPoolExecutor with one thread makes it equivalent to Timer."


- Antal Sinkovits


On okt. 20, 2018, 7:13 du, Bharathkrishna Guruvayoor Murali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69107/
> ---
> 
> (Updated okt. 20, 2018, 7:13 du)
> 
> 
> Review request for hive, Antal Sinkovits, Sahil Takiar, and Vihang 
> Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Improve record and memory usage logging in SparkRecordHandler
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 
> 88dd12c05ade417aca4cdaece4448d31d4e1d65f 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMergeFileRecordHandler.java
>  8880bb604e088755dcfb0bcb39689702fab0cb77 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkRecordHandler.java 
> cb5bd7ada2d5ad4f1f654cf80ddaf4504be5d035 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkReduceRecordHandler.java
>  20e7ea0f4e8d4ff79dddeaab0406fc7350d22bd7 
> 
> 
> Diff: https://reviews.apache.org/r/69107/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bharathkrishna Guruvayoor Murali
> 
>

[jira] [Created] (HIVE-20794) Use Zookeeper for metastore service discovery

2018-10-24 Thread Ashutosh Bapat (JIRA) 
Ashutosh Bapat created HIVE-20794:
-

 Summary: Use Zookeeper for metastore service discovery
 Key: HIVE-20794
 URL: https://issues.apache.org/jira/browse/HIVE-20794
 Project: Hive
  Issue Type: Improvement
Reporter: Ashutosh Bapat
Assignee: Ashutosh Bapat


Right now, multiple metastore services can be specified in hive.metastore.uris 
configuration, but that list is static and can not be modified dynamically. Use 
Zookeeper for dynamic service discovery of metastore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)