[
https://issues.apache.org/jira/browse/HIVE-23710?focusedWorklogId=476929&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476929
]
ASF GitHub Bot logged work on HIVE-23710:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Sep/20 00:46
Start Date: 01/Sep/20 00:46
Worklog Time Spent: 10m
Work Description: github-actions[bot] commented on pull request #1135:
URL: https://github.com/apache/hive/pull/1135#issuecomment-684124148
This pull request has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the d...@hive.apache.org list if the patch is in
need of reviews.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 476929)
Time Spent: 20m (was: 10m)
> Add table meta cache limit when starting Hive server2
> -----------------------------------------------------
>
> Key: HIVE-23710
> URL: https://issues.apache.org/jira/browse/HIVE-23710
> Project: Hive
> Issue Type: Improvement
> Environment: Hive 2.3.6
> Reporter: Deegue
> Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23710.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When we start up Hive server2, it will connect to metastore to get table meta
> info by database and cache them. If there are many tables in a database,
> however, will exceed `hive.metastore.client.socket.timeout`.
> Then exception thrown like:
> {noformat}
> 2020-06-17T11:38:27,595 WARN [main] metastore.RetryingMetaStoreClient:
> MetaStoreClient lost connection. Attempting to reconnect (1 of 1) after 1s.
> getTableObjectsByName
> org.apache.thrift.transport.TTransportException:
> java.net.SocketTimeoutException: Read timed out
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_objects_by_name_req(ThriftHiveMetastore.java:1596)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_objects_by_name_req(ThriftHiveMetastore.java:1583)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableObjectsByName(HiveMetaStoreClient.java:1370)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTableObjectsByName(SessionHiveMetaStoreClient.java:238)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:206)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at com.sun.proxy.$Proxy38.getTableObjectsByName(Unknown Source) ~[?:?]
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2336)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at com.sun.proxy.$Proxy38.getTableObjectsByName(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getAllTableObjects(Hive.java:1343)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.hadoop.hive.ql.metadata.HiveMaterializedViewsRegistry.init(HiveMaterializedViewsRegistry.java:127)
> ~[hive-exec-2.3.6.jar:2.3.6]
> at
> org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:167)
> ~[hive-service-2.3.6.jar:2.3.6]
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:607)
> ~[hive-service-2.3.6.jar:2.3.6]
> at
> org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:100)
> ~[hive-service-2.3.6.jar:2.3.6]
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:855)
> ~[hive-service-2.3.6.jar:2.3.6]
> at
> org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:724)
> ~[hive-service-2.3.6.jar:2.3.6]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_121]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_121]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
> ~[hadoop-common-2.6.0-cdh5.16.1.jar:?]
> at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
> ~[hadoop-common-2.6.0-cdh5.16.1.jar:?]
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_121]
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> ~[?:1.8.0_121]
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> ~[?:1.8.0_121]
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> ~[?:1.8.0_121]
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> ~[?:1.8.0_121]
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> ~[?:1.8.0_121]
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> ~[hive-exec-2.3.6.jar:2.3.6]
> ... 32 more
> {noformat}
> The solution is add `hive.server2.init.load.table.limit` to control max
> number of tables for one TCP request from Hive server2 to metastore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
