[ https://issues.apache.org/jira/browse/HIVE-27773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HIVE-27773 started by Ramesh Kumar Thangarajan. ------------------------------------------------------- > get_valid_write_ids is being called multiple times for a single query > --------------------------------------------------------------------- > > Key: HIVE-27773 > URL: https://issues.apache.org/jira/browse/HIVE-27773 > Project: Hive > Issue Type: Task > Reporter: Ramesh Kumar Thangarajan > Assignee: Ramesh Kumar Thangarajan > Priority: Major > > Looking at the below logs suggest that the get_valid_write_ids is not cached > for a single query for a single table. It is being called multiple times > across different phases in the compilation of the query. We should verify if > we can safely cache and re use the results. That way we can avoid around > 40-50 ms out of 678ms compilation time. > > {code:java} > 2023-09-19T02:55:06,940 INFO [fa0fa087-7e2c-45b8-bd27-b94fbbe23e49 Listener > at 0.0.0.0/50501] ql.Driver: Compiling > command(queryId=rameshkumar_20230919025506_b005cc57-1717-4798-b8da-b502aa7ca3d6): > 2023-09-19T02:55:06,967 DEBUG [fa0fa087-7e2c-45b8-bd27-b94fbbe23e49 Listener > at 0.0.0.0/50501] metrics.PerfLogger: <PERFLOG method=get_valid_write_ids > from=org.apache.hadoop.hive.metastore.RetryingHMSHandler> > 2023-09-19T02:55:06,979 DEBUG [fa0fa087-7e2c-45b8-bd27-b94fbbe23e49 Listener > at 0.0.0.0/50501] metrics.PerfLogger: </PERFLOG method=get_valid_write_ids > start=1695117306967 end=1695117306979 duration=12 > from=org.apache.hadoop.hive.metastore.RetryingHMSHandler retryCount=0 > error=false> > 2023-09-19T02:55:06,980 DEBUG [fa0fa087-7e2c-45b8-bd27-b94fbbe23e49 Listener > at 0.0.0.0/50501] metrics.PerfLogger: <PERFLOG method=get_valid_write_ids > from=org.apache.hadoop.hive.metastore.RetryingHMSHandler> > 2023-09-19T02:55:06,986 DEBUG [fa0fa087-7e2c-45b8-bd27-b94fbbe23e49 Listener > at 0.0.0.0/50501] metrics.PerfLogger: </PERFLOG method=get_valid_write_ids > start=1695117306980 end=1695117306986 duration=6 > from=org.apache.hadoop.hive.metastore.RetryingHMSHandler retryCount=0 > error=false> > 2023-09-19T02:55:06,988 DEBUG [fa0fa087-7e2c-45b8-bd27-b94fbbe23e49 Listener > at 0.0.0.0/50501] metrics.PerfLogger: <PERFLOG method=get_valid_write_ids > from=org.apache.hadoop.hive.metastore.RetryingHMSHandler> > 2023-09-19T02:55:06,995 DEBUG [fa0fa087-7e2c-45b8-bd27-b94fbbe23e49 Listener > at 0.0.0.0/50501] metrics.PerfLogger: </PERFLOG method=get_valid_write_ids > start=1695117306988 end=1695117306995 duration=7 > from=org.apache.hadoop.hive.metastore.RetryingHMSHandler retryCount=0 > error=false> > 2023-09-19T02:55:06,997 DEBUG [fa0fa087-7e2c-45b8-bd27-b94fbbe23e49 Listener > at 0.0.0.0/50501] metrics.PerfLogger: <PERFLOG method=get_valid_write_ids > from=org.apache.hadoop.hive.metastore.RetryingHMSHandler> > 2023-09-19T02:55:07,007 DEBUG [fa0fa087-7e2c-45b8-bd27-b94fbbe23e49 Listener > at 0.0.0.0/50501] metrics.PerfLogger: </PERFLOG method=get_valid_write_ids > start=1695117306997 end=1695117307007 duration=10 > from=org.apache.hadoop.hive.metastore.RetryingHMSHandler retryCount=0 > error=false> > 2023-09-19T02:55:07,009 DEBUG [fa0fa087-7e2c-45b8-bd27-b94fbbe23e49 Listener > at 0.0.0.0/50501] metrics.PerfLogger: <PERFLOG method=get_valid_write_ids > from=org.apache.hadoop.hive.metastore.RetryingHMSHandler> > 2023-09-19T02:55:07,017 DEBUG [fa0fa087-7e2c-45b8-bd27-b94fbbe23e49 Listener > at 0.0.0.0/50501] metrics.PerfLogger: </PERFLOG method=get_valid_write_ids > start=1695117307009 end=1695117307017 duration=8 > from=org.apache.hadoop.hive.metastore.RetryingHMSHandler retryCount=0 > error=false> > 2023-09-19T02:55:07,018 DEBUG [fa0fa087-7e2c-45b8-bd27-b94fbbe23e49 Listener > at 0.0.0.0/50501] metrics.PerfLogger: <PERFLOG method=get_valid_write_ids > from=org.apache.hadoop.hive.metastore.RetryingHMSHandler> > 2023-09-19T02:55:07,026 DEBUG [fa0fa087-7e2c-45b8-bd27-b94fbbe23e49 Listener > at 0.0.0.0/50501] metrics.PerfLogger: </PERFLOG method=get_valid_write_ids > start=1695117307018 end=1695117307026 duration=8 > from=org.apache.hadoop.hive.metastore.RetryingHMSHandler retryCount=0 > error=false> > 2023-09-19T02:55:07,059 DEBUG [fa0fa087-7e2c-45b8-bd27-b94fbbe23e49 Listener > at 0.0.0.0/50501] metrics.PerfLogger: <PERFLOG method=get_valid_write_ids > from=org.apache.hadoop.hive.metastore.RetryingHMSHandler> > 2023-09-19T02:55:07,068 DEBUG [fa0fa087-7e2c-45b8-bd27-b94fbbe23e49 Listener > at 0.0.0.0/50501] metrics.PerfLogger: </PERFLOG method=get_valid_write_ids > start=1695117307059 end=1695117307068 duration=9 > from=org.apache.hadoop.hive.metastore.RetryingHMSHandler retryCount=0 > error=false> > 2023-09-19T02:55:07,618 INFO [fa0fa087-7e2c-45b8-bd27-b94fbbe23e49 Listener > at 0.0.0.0/50501] ql.Driver: Completed compiling > command(queryId=rameshkumar_20230919025506_b005cc57-1717-4798-b8da-b502aa7ca3d6); > Time taken: 0.678 seconds{code} -- This message was sent by Atlassian Jira (v8.20.10#820010)