Rajesh Balamohan created HIVE-15377:
---------------------------------------
Summary: Driver::acquireWriteIds can be expensive trying to get
details from MS
Key: HIVE-15377
URL: https://issues.apache.org/jira/browse/HIVE-15377
Project: Hive
Issue Type: Sub-task
Affects Versions: hive-14535
Reporter: Rajesh Balamohan
Branch: hive-14535
Populated data in tpc-ds web_returns table. Select queries take longer time
trying to acquire writeIds from MS.
{noformat}
hive> select * from web_returns_hive_commit limit 10;
select * from web_returns_hive_commit limit 10
...
....
Time taken: 52.494 seconds, Fetched: 10 row(s)
{noformat}
Without commit feature, same query would execute in ~6 seconds.
Attaching the stacktrace for reference:
{noformat}
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at
org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143)
at
org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112)
at
org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:70)
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:283)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1799)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:200)
- locked <0x0000000223192988> (a
org.postgresql.core.v3.QueryExecutorImpl)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:424)
at
org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:161)
at
org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:114)
at
com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:174)
at
org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeQuery(ParamLoggingPreparedStatement.java:375)
at
org.datanucleus.store.rdbms.SQLController.executeStatementQuery(SQLController.java:552)
at
org.datanucleus.store.rdbms.scostore.ElementContainerStore.getSize(ElementContainerStore.java:660)
at
org.datanucleus.store.rdbms.scostore.ElementContainerStore.size(ElementContainerStore.java:606)
at org.datanucleus.store.types.wrappers.backed.List.size(List.java:542)
- locked <0x000000078e8b4f60> (a
org.datanucleus.store.types.wrappers.backed.List)
at
org.apache.hadoop.hive.metastore.ObjectStore.convertToOrders(ObjectStore.java:1665)
at
org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1710)
at
org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1725)
at
org.apache.hadoop.hive.metastore.ObjectStore.convertToTable(ObjectStore.java:1578)
at
org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:1274)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
at com.sun.proxy.$Proxy47.getTable(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_valid_write_ids(HiveMetaStore.java:6874)
at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
at com.sun.proxy.$Proxy50.get_valid_write_ids(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getValidWriteIds(HiveMetaStoreClient.java:2480)
at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:162)
at com.sun.proxy.$Proxy51.getValidWriteIds(Unknown Source)
at
org.apache.hadoop.hive.ql.metadata.Hive.getValidWriteIdsForTable(Hive.java:4173)
at org.apache.hadoop.hive.ql.Driver.acquireWriteIds(Driver.java:1592)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1517)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1226)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1216)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:400)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:777)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:715)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:642)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
