[jira] [Created] (DRILL-5738) Drill query takes 10+ minutes before start executing, excessive Hive metastore queries
kevin zou created DRILL-5738: Summary: Drill query takes 10+ minutes before start executing, excessive Hive metastore queries Key: DRILL-5738 URL: https://issues.apache.org/jira/browse/DRILL-5738 Project: Apache Drill Issue Type: Bug Components: Query Planning & Optimization Affects Versions: 1.6.0 Environment: mapr 5.2 Reporter: kevin zou Priority: Critical I ve a Drill query on 14 tables in Hive. The query took a few seconds to execute. However, the query would stay in "Starting" state for 10+ minutes before execution. I set up the log to "Debug" mode to figure out what Drill had been doing during the 10+ minutes, only to find out Drill generated excessive number of meta data queries to hive meta store. Although each query took a few micro seconds (meta data cached in memory), the number of queries was 3438793. drillbit.log:2017-06-05 18:50:57,201 [26ca5bda-5e87-475a-cd93-17c6957cc3ee:foreman] DEBUG o.a.d.e.s.hive.HiveMetadataProvider - Took 4 µs to get stats from idm_intel_1x.lu_jde_emp_directory drillbit.log:2017-06-05 18:50:57,201 [26ca5bda-5e87-475a-cd93-17c6957cc3ee:foreman] DEBUG o.a.drill.exec.store.hive.HiveScan - HiveStats: numRows: 15, sizeInBytes: 15 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5738) Drill query takes 10+ minutes before start executing, excessive Hive metastore queries
[ https://issues.apache.org/jira/browse/DRILL-5738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kevin zou updated DRILL-5738: - Affects Version/s: 1.10.0 > Drill query takes 10+ minutes before start executing, excessive Hive > metastore queries > -- > > Key: DRILL-5738 > URL: https://issues.apache.org/jira/browse/DRILL-5738 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Affects Versions: 1.6.0, 1.10.0 > Environment: mapr 5.2 >Reporter: kevin zou >Priority: Critical > > I ve a Drill query on 14 tables in Hive. The query took a few seconds to > execute. However, the query would stay in "Starting" state for 10+ minutes > before execution. > > I set up the log to "Debug" mode to figure out what Drill had been doing > during the 10+ minutes, only to find out Drill generated excessive number of > meta data queries to hive meta store. > > Although each query took a few micro seconds (meta data cached in memory), > the number of queries was 3438793. > drillbit.log:2017-06-05 18:50:57,201 > [26ca5bda-5e87-475a-cd93-17c6957cc3ee:foreman] DEBUG > o.a.d.e.s.hive.HiveMetadataProvider > - Took 4 µs to get stats from idm_intel_1x.lu_jde_emp_directory > drillbit.log:2017-06-05 18:50:57,201 > [26ca5bda-5e87-475a-cd93-17c6957cc3ee:foreman] DEBUG > o.a.drill.exec.store.hive.HiveScan > - HiveStats: numRows: 15, sizeInBytes: 15 > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5738) Drill query takes 10+ minutes before start executing, excessive Hive metastore queries
[ https://issues.apache.org/jira/browse/DRILL-5738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kevin zou updated DRILL-5738: - Attachment: log.tar.gz Logs in debug mode > Drill query takes 10+ minutes before start executing, excessive Hive > metastore queries > -- > > Key: DRILL-5738 > URL: https://issues.apache.org/jira/browse/DRILL-5738 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Affects Versions: 1.6.0, 1.10.0 > Environment: mapr 5.2 >Reporter: kevin zou >Priority: Critical > Attachments: log.tar.gz > > > I ve a Drill query on 14 tables in Hive. The query took a few seconds to > execute. However, the query would stay in "Starting" state for 10+ minutes > before execution. > > I set up the log to "Debug" mode to figure out what Drill had been doing > during the 10+ minutes, only to find out Drill generated excessive number of > meta data queries to hive meta store. > > Although each query took a few micro seconds (meta data cached in memory), > the number of queries was 3438793. > drillbit.log:2017-06-05 18:50:57,201 > [26ca5bda-5e87-475a-cd93-17c6957cc3ee:foreman] DEBUG > o.a.d.e.s.hive.HiveMetadataProvider > - Took 4 µs to get stats from idm_intel_1x.lu_jde_emp_directory > drillbit.log:2017-06-05 18:50:57,201 > [26ca5bda-5e87-475a-cd93-17c6957cc3ee:foreman] DEBUG > o.a.drill.exec.store.hive.HiveScan > - HiveStats: numRows: 15, sizeInBytes: 15 > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5738) Drill query takes 10+ minutes before start executing, excessive Hive metastore queries
[ https://issues.apache.org/jira/browse/DRILL-5738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149145#comment-16149145 ] kevin zou commented on DRILL-5738: -- on 1.10, the number of metadata query reduced from millions to thousands. But overall time to spend for sql path generation is still very high (7 minutes) > Drill query takes 10+ minutes before start executing, excessive Hive > metastore queries > -- > > Key: DRILL-5738 > URL: https://issues.apache.org/jira/browse/DRILL-5738 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Affects Versions: 1.6.0, 1.10.0 > Environment: mapr 5.2 >Reporter: kevin zou >Priority: Critical > Attachments: log.tar.gz > > > I ve a Drill query on 14 tables in Hive. The query took a few seconds to > execute. However, the query would stay in "Starting" state for 10+ minutes > before execution. > > I set up the log to "Debug" mode to figure out what Drill had been doing > during the 10+ minutes, only to find out Drill generated excessive number of > meta data queries to hive meta store. > > Although each query took a few micro seconds (meta data cached in memory), > the number of queries was 3438793. > drillbit.log:2017-06-05 18:50:57,201 > [26ca5bda-5e87-475a-cd93-17c6957cc3ee:foreman] DEBUG > o.a.d.e.s.hive.HiveMetadataProvider > - Took 4 µs to get stats from idm_intel_1x.lu_jde_emp_directory > drillbit.log:2017-06-05 18:50:57,201 > [26ca5bda-5e87-475a-cd93-17c6957cc3ee:foreman] DEBUG > o.a.drill.exec.store.hive.HiveScan > - HiveStats: numRows: 15, sizeInBytes: 15 > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
