Hi, I am running into this problem again. The solution is to define a view that is a "union all" of individual select queries of single csv files. Can I request that the hardcoded timeout values be parametrized please?
Herman On Tue, Oct 2, 2018 at 7:29 PM Herman Tan <[email protected]> wrote: > Hi, > > I found the hard coded parameter in the source code > > > https://github.com/apache/drill/blob/8edeb49873d1a1710cfe28e0b49364d07eb1aef4/exec/java-exec/src/main/java/org/apache/drill/exec/store/TimedCallable.java > > LINE 52 : private static long TIMEOUT_PER_RUNNABLE_IN_MSECS = 15000; > LINE 210 :timeout = TIMEOUT_PER_RUNNABLE_IN_MSECS * ((tasks.size() - > 1)/parallelism + 1); > > parallelism param is also hardcoded > > https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/metadata/Metadata.java > LINE 343: 16 > > In my case: task size is 29, 15000 * ((29-1)/(16+1)) = 15000 * 2 = 30000 > 16 runnables executes round robbin on 29 tasks. Each runnable given 30000 > ms to timeout. > > This is the error message: > Waited for 30000 ms, but only 11 tasks for 'Fetch parquet metadata' are > complete. Total number of tasks 29, parallelism 16. > > TimedCallable.java: > LINE 248: final String errMsg = String.format("Waited for %d ms, but > only %d tasks for '%s' are complete." + > " Total number of tasks %d, parallelism %d.", timeout, > futureMapper.count, activity, tasks.size(), parallelism); > > Shouldn't these be parameterized in "options" based on the infrastructure? > > Regards, > Herman > > > > > On Tue, Oct 2, 2018 at 6:47 PM Herman Tan <[email protected]> wrote: > >> Hi, >> >> I have restarted drill and run the script again. >> >> select * from dfs.tmp.`load_pos_sales_detail_tbl`; >> -- SQL Error: RESOURCE ERROR: Waited for 30000 ms, but only 11 tasks for >> 'Fetch parquet metadata' are complete. Total number of tasks 29, >> parallelism 16. >> >> The 29 tasks is related to the 29 parquet files in the folder. >> To check if any of the parquet files has an error, I ran the following >> SQL on each parquet file in the folder. ALL PASSED. (SQL Below). >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_0_0.parquet`; >> >> So it seems that for this table drill can only get the metadata for 11 >> parquet files before it times out. >> The time-out is a calculation and it varies from size of table. >> I checked the source code but I cannot find the calculation of the >> timeout of "30000 ms". >> When I am lucky, drill can resolve the metadata for 29 files in 30000 ms >> and it passes. >> >> I plan to use drill for production but this bothers me that there is a >> limit on the number of parquet files and the timeout parameter cannot be >> tuned. >> >> Does anyone have any ideas? >> >> Regards, >> Herman >> -------------- SQL BELOW ------------- >> >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_0_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_10_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_10_1.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_11_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_11_1.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_12_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_12_1.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_13_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_13_1.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_14_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_15_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_15_1.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_16_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_16_1.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_1_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_2_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_3_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_4_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_4_1.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_5_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_5_1.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_6_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_6_1.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_7_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_7_1.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_8_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_8_1.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_9_0.parquet`; >> select * from dfs.tmp.`load_pos_sales_detail_tbl/1_9_1.parquet`; >> >> >> On Tue, Oct 2, 2018 at 4:44 PM Herman Tan <[email protected]> wrote: >> >>> Hi Divya and everyone, >>> >>> The problem has disappeared. >>> Drill was not restarted. >>> This appears to be intermittent. >>> Before I submitted the error report, I ran the script several times and >>> it failed all the time. >>> Today I ran it again and it succeeded. >>> I will restart and test again. >>> >>> Regards, >>> Herman >>> >>> >>> >>> On Thu, Sep 27, 2018 at 11:50 AM Divya Gehlot <[email protected]> >>> wrote: >>> >>>> Hi Herman, >>>> Just to ensure that your parquet file format is not corrupted , Can you >>>> please query a folder like just 2001 or some of the files underneath >>>> .Instead of querying the whole data set at once . >>>> >>>> Thanks, >>>> Divya >>>> >>>> On Wed, 26 Sep 2018 at 15:35, Herman Tan <[email protected]> wrote: >>>> >>>> > Hi Kunal, >>>> > >>>> > ---- >>>> > That said, could you provide some details about the parquet data >>>> you've >>>> > created, like the schema, parquet version and the tool used to >>>> generate. >>>> > Usually, the schema (and meta) provides most of these details for any >>>> > parquet file. >>>> > ---- >>>> > >>>> > 1. The schema is under dfs.tmp, the queries to generate are all >>>> documented >>>> > below. >>>> > 2. I don't know how to find the parquet version of the data file >>>> > 3. The tool used to generate the parquest is apache drill. The CTAS >>>> is >>>> > detailed below. >>>> > >>>> > Regards, >>>> > Herman >>>> > ____________ >>>> > >>>> > *This is the Text data* >>>> > >>>> > This is the folders of the files >>>> > Total # of lines about 50 million rows >>>> > ---------- >>>> > show files from >>>> dfs.`D:\retail_sandbox\pos\sales_pos_detail\pos_details_ >>>> > 20180825` >>>> > ; >>>> > show files from >>>> dfs.`D:\retail_sandbox\pos\sales_pos_detail\pos_details_ >>>> > 20180825\2011` >>>> > ; >>>> > ----- >>>> > sales_pos_detail >>>> > \pos_details_20180825 >>>> > \2007 >>>> > \2008 >>>> > \2009 >>>> > \2010 >>>> > \2011 >>>> > \pos_details_0.csv >>>> > \pos_details_1.csv >>>> > \pos_details_2.csv >>>> > \pos_details_3.csv >>>> > \pos_details_4.csv >>>> > \pos_details_5.csv >>>> > \pos_details_6.csv >>>> > \pos_details_7.csv >>>> > \pos_details_8.csv >>>> > \2012 >>>> > \2013 >>>> > \2014 >>>> > \2015 >>>> > \2016 >>>> > \2017 >>>> > \2018 >>>> > \others >>>> > ----- >>>> > >>>> > *This is the view with the metadata defined:* >>>> > >>>> > create or replace view dfs.tmp.load_pos_sales_detail as >>>> > SELECT >>>> > -- dimension keys >>>> > cast(dim_date_key as int) dim_date_key >>>> > ,cast(dim_site_key as int) dim_site_key >>>> > ,cast(dim_pos_header_key as bigint) dim_pos_header_key >>>> > ,cast(dim_pos_cashier_key as int) dim_pos_cashier_key >>>> > ,cast(dim_card_number_key as int) dim_card_number_key >>>> > ,cast(dim_hour_minute_key as int) dim_hour_minute_key >>>> > ,cast(dim_pos_clerk_key as int) dim_pos_clerk_key >>>> > ,cast(dim_product_key as int) dim_product_key >>>> > ,cast(dim_pos_employee_purchase_key as int) >>>> dim_pos_employee_purchase_key >>>> > ,cast(dim_pos_terminal_key as int) dim_pos_terminal_key >>>> > ,cast(dim_campaign_key as int) dim_campaign_key >>>> > ,cast(dim_promo_key as int) dim_promo_key >>>> > ,cast( case when dim_site_lfl_key = '' then 0 else dim_site_lfl_key >>>> end as >>>> > int) dim_site_lfl_key >>>> > -- derived from keys >>>> > ,dim_date_str >>>> > ,`year` as `trx_year` >>>> > -- Measures >>>> > ,Product_Sales_Qty >>>> > ,Product_Sales_Price >>>> > ,Product_Cost_Price >>>> > ,Product_Cost_Amt >>>> > ,Product_Sales_Gross_Amt >>>> > ,Product_Sales_Promo_Disc_Amt >>>> > ,Product_Sales_Add_Promo_Disc_Amt >>>> > ,Product_Sales_Total_Promo_Disc_Amt >>>> > ,Product_Sales_Retail_Promo_Amt >>>> > ,Product_Sales_Retail_Amt >>>> > ,Product_Sales_VAT_Amt >>>> > ,Product_Sales_Product_Margin_Amt >>>> > ,Product_Sales_Initial_Margin_Amt >>>> > from dfs.`D:\retail_sandbox\pos\sales_pos_detail\pos_details_20180825` >>>> > ; >>>> > >>>> > >>>> > *This is the CTAS that generates the parquet from the view above:* >>>> > >>>> > drop table if exists dfs.tmp.load_pos_sales_detail_tbl >>>> > ; >>>> > >>>> > create table dfs.tmp.load_pos_sales_detail_tbl AS >>>> > SELECT >>>> > -- dimension keys >>>> > dim_date_key >>>> > ,dim_site_key >>>> > ,dim_pos_header_key >>>> > ,dim_pos_cashier_key >>>> > ,dim_card_number_key >>>> > ,dim_hour_minute_key >>>> > ,dim_pos_clerk_key >>>> > ,dim_product_key >>>> > ,dim_pos_employee_purchase_key >>>> > ,dim_pos_terminal_key >>>> > ,dim_campaign_key >>>> > ,dim_promo_key >>>> > ,dim_site_lfl_key >>>> > -- derived from keys >>>> > ,dim_date_str >>>> > ,`trx_year` >>>> > -- Measures >>>> > ,Product_Sales_Qty Sales_Qty >>>> > ,Product_Sales_Price Sales_Price >>>> > ,Product_Cost_Price Cost_Price >>>> > ,Product_Cost_Amt Cost_Amt >>>> > ,Product_Sales_Gross_Amt Sales_Gross_Amt >>>> > ,Product_Sales_Promo_Disc_Amt Sales_Promo_Disc_Amt >>>> > ,Product_Sales_Add_Promo_Disc_Amt Add_Promo_Disc_Amt >>>> > ,Product_Sales_Total_Promo_Disc_Amt Total_Promo_Disc_Amt >>>> > ,Product_Sales_Retail_Promo_Amt Retail_Promo_Amt >>>> > ,Product_Sales_Retail_Amt Retail_Amt >>>> > ,Product_Sales_VAT_Amt VAT_Amt >>>> > ,Product_Sales_Product_Margin_Amt Product_Margin_Amt >>>> > ,Product_Sales_Initial_Margin_Amt Initial_Margin_Amt >>>> > from dfs.tmp.load_pos_sales_detail >>>> > ; >>>> > >>>> > >>>> > *This is the select query that generated the error:* >>>> > >>>> > select * >>>> > from dfs.tmp.load_pos_sales_detail_tbl >>>> > ; >>>> > >>>> > ----- ERROR ---------------------------- >>>> > >>>> > SQL Error: RESOURCE ERROR: Waited for 30000 ms, but only 10 tasks for >>>> > 'Fetch parquet metadata' are complete. Total number of tasks 29, >>>> > parallelism 16. >>>> > >>>> > >>>> > On Mon, Sep 24, 2018 at 9:08 AM, Kunal Khatua <[email protected]> >>>> wrote: >>>> > >>>> > > Hi Herman >>>> > > >>>> > > Assuming that you're doing analytics on your data. If that's the >>>> case, >>>> > > parquet format is the way to go. >>>> > > >>>> > > That said, could you provide some details about the parquet data >>>> you've >>>> > > created, like the schema, parquet version and the tool used to >>>> generate. >>>> > > Usually, the schema (and meta) provides most of these details for >>>> any >>>> > > parquet file. >>>> > > >>>> > > It'll be useful to know if there is a pattern in the failure >>>> because of >>>> > > which there might be corruption occurring. >>>> > > >>>> > > Kunal >>>> > > >>>> > > >>>> > > On 9/22/2018 11:49:36 PM, Herman Tan <[email protected]> wrote: >>>> > > Hi Karthik, >>>> > > >>>> > > Thank you for pointing me to the mail archive in May 2018. >>>> > > That is exactly the same problem I am facing. >>>> > > >>>> > > I thought of using Drill as an ETL where I load the warehouse >>>> parquet >>>> > > tables from text source files. >>>> > > Then I query the parquet tables. >>>> > > It works on some parquet tables but am having problems with large >>>> ones >>>> > that >>>> > > consist of several files. (I think) >>>> > > Still investigating. >>>> > > Anyone in the community have other experience? >>>> > > Should I work with all text files instead of parquet? >>>> > > >>>> > > >>>> > > Herman >>>> > > >>>> > > >>>> > > On Fri, Sep 21, 2018 at 2:15 AM, Karthikeyan Manivannan >>>> > > [email protected]> wrote: >>>> > > >>>> > > > Hi Herman, >>>> > > > >>>> > > > I am not sure what the exact problem here is but can you check to >>>> see >>>> > if >>>> > > > you are not hitting the problem described here: >>>> > > > >>>> > > > http://mail-archives.apache.org/mod_mbox/drill-user/201805.mbox/% >>>> > > > >>>> 3ccacwrgnexlxop2vcyugsa4gwd1jgs8f+rcpzq8rhuatfw5fm...@mail.gmail.com >>>> > %3E >>>> > > > >>>> > > > Thanks >>>> > > > >>>> > > > Karthik >>>> > > > >>>> > > > On Wed, Sep 19, 2018 at 7:02 PM Herman Tan wrote: >>>> > > > >>>> > > > > Hi, >>>> > > > > >>>> > > > > I encountered the following error. >>>> > > > > The Steps I did are as follows: >>>> > > > > 1. Create a view to fix the data type of fields with cast >>>> > > > > 2. Create table (parquet) using the view >>>> > > > > 3. Query select * from table (query a field also does not work) >>>> > > > > >>>> > > > > The error: >>>> > > > > SQL Error: RESOURCE ERROR: Waited for 30000 ms, but only 10 >>>> tasks for >>>> > > > > 'Fetch parquet metadata' are complete. Total number of tasks 29, >>>> > > > > parallelism 16. >>>> > > > > >>>> > > > > When I re-run this, the number of tasks will vary. >>>> > > > > >>>> > > > > What could be the problem? >>>> > > > > >>>> > > > > Regards, >>>> > > > > Herman Tan >>>> > > > > >>>> > > > > More info below: >>>> > > > > >>>> > > > > This is the folders of the files >>>> > > > > Total # of lines, 50 million >>>> > > > > ---------- >>>> > > > > show files from >>>> > > > > >>>> dfs.`D:\retail_sandbox\pos\sales_pos_detail\pos_details_20180825` >>>> > > > > ; >>>> > > > > show files from >>>> > > > > >>>> > dfs.`D:\retail_sandbox\pos\sales_pos_detail\pos_details_20180825\2011` >>>> > > > > ; >>>> > > > > ----- >>>> > > > > sales_pos_detail >>>> > > > > \pos_details_20180825 >>>> > > > > \2007 >>>> > > > > \2008 >>>> > > > > \2009 >>>> > > > > \2010 >>>> > > > > \2011 >>>> > > > > \pos_details_0.csv >>>> > > > > \pos_details_1.csv >>>> > > > > \pos_details_2.csv >>>> > > > > \pos_details_3.csv >>>> > > > > \pos_details_4.csv >>>> > > > > \pos_details_5.csv >>>> > > > > \pos_details_6.csv >>>> > > > > \pos_details_7.csv >>>> > > > > \pos_details_8.csv >>>> > > > > \2012 >>>> > > > > \2013 >>>> > > > > \2014 >>>> > > > > \2015 >>>> > > > > \2016 >>>> > > > > \2017 >>>> > > > > \2018 >>>> > > > > \others >>>> > > > > ----- >>>> > > > > >>>> > > > > create or replace view dfs.tmp.load_pos_sales_detail as >>>> > > > > SELECT >>>> > > > > -- dimension keys >>>> > > > > cast(dim_date_key as int) dim_date_key >>>> > > > > ,cast(dim_site_key as int) dim_site_key >>>> > > > > ,cast(dim_pos_header_key as bigint) dim_pos_header_key >>>> > > > > ,cast(dim_pos_cashier_key as int) dim_pos_cashier_key >>>> > > > > ,cast(dim_card_number_key as int) dim_card_number_key >>>> > > > > ,cast(dim_hour_minute_key as int) dim_hour_minute_key >>>> > > > > ,cast(dim_pos_clerk_key as int) dim_pos_clerk_key >>>> > > > > ,cast(dim_product_key as int) dim_product_key >>>> > > > > ,cast(dim_pos_employee_purchase_key as int) >>>> > > > dim_pos_employee_purchase_key >>>> > > > > ,cast(dim_pos_terminal_key as int) dim_pos_terminal_key >>>> > > > > ,cast(dim_campaign_key as int) dim_campaign_key >>>> > > > > ,cast(dim_promo_key as int) dim_promo_key >>>> > > > > ,cast( case when dim_site_lfl_key = '' then 0 else >>>> dim_site_lfl_key >>>> > end >>>> > > > as >>>> > > > > int) dim_site_lfl_key >>>> > > > > -- derived from keys >>>> > > > > ,dim_date_str >>>> > > > > ,`year` as `trx_year` >>>> > > > > -- Measures >>>> > > > > ,Product_Sales_Qty >>>> > > > > ,Product_Sales_Price >>>> > > > > ,Product_Cost_Price >>>> > > > > ,Product_Cost_Amt >>>> > > > > ,Product_Sales_Gross_Amt >>>> > > > > ,Product_Sales_Promo_Disc_Amt >>>> > > > > ,Product_Sales_Add_Promo_Disc_Amt >>>> > > > > ,Product_Sales_Total_Promo_Disc_Amt >>>> > > > > ,Product_Sales_Retail_Promo_Amt >>>> > > > > ,Product_Sales_Retail_Amt >>>> > > > > ,Product_Sales_VAT_Amt >>>> > > > > ,Product_Sales_Product_Margin_Amt >>>> > > > > ,Product_Sales_Initial_Margin_Amt >>>> > > > > from >>>> > dfs.`D:\retail_sandbox\pos\sales_pos_detail\pos_details_20180825` >>>> > > > > ; >>>> > > > > >>>> > > > > drop table if exists dfs.tmp.load_pos_sales_detail_tbl >>>> > > > > ; >>>> > > > > >>>> > > > > create table dfs.tmp.load_pos_sales_detail_tbl AS >>>> > > > > SELECT >>>> > > > > -- dimension keys >>>> > > > > dim_date_key >>>> > > > > ,dim_site_key >>>> > > > > ,dim_pos_header_key >>>> > > > > ,dim_pos_cashier_key >>>> > > > > ,dim_card_number_key >>>> > > > > ,dim_hour_minute_key >>>> > > > > ,dim_pos_clerk_key >>>> > > > > ,dim_product_key >>>> > > > > ,dim_pos_employee_purchase_key >>>> > > > > ,dim_pos_terminal_key >>>> > > > > ,dim_campaign_key >>>> > > > > ,dim_promo_key >>>> > > > > ,dim_site_lfl_key >>>> > > > > -- derived from keys >>>> > > > > ,dim_date_str >>>> > > > > ,`trx_year` >>>> > > > > -- Measures >>>> > > > > ,Product_Sales_Qty Sales_Qty >>>> > > > > ,Product_Sales_Price Sales_Price >>>> > > > > ,Product_Cost_Price Cost_Price >>>> > > > > ,Product_Cost_Amt Cost_Amt >>>> > > > > ,Product_Sales_Gross_Amt Sales_Gross_Amt >>>> > > > > ,Product_Sales_Promo_Disc_Amt Sales_Promo_Disc_Amt >>>> > > > > ,Product_Sales_Add_Promo_Disc_Amt Add_Promo_Disc_Amt >>>> > > > > ,Product_Sales_Total_Promo_Disc_Amt Total_Promo_Disc_Amt >>>> > > > > ,Product_Sales_Retail_Promo_Amt Retail_Promo_Amt >>>> > > > > ,Product_Sales_Retail_Amt Retail_Amt >>>> > > > > ,Product_Sales_VAT_Amt VAT_Amt >>>> > > > > ,Product_Sales_Product_Margin_Amt Product_Margin_Amt >>>> > > > > ,Product_Sales_Initial_Margin_Amt Initial_Margin_Amt >>>> > > > > from dfs.tmp.load_pos_sales_detail >>>> > > > > ; >>>> > > > > >>>> > > > > select * >>>> > > > > from dfs.tmp.load_pos_sales_detail_tbl >>>> > > > > ; >>>> > > > > >>>> > > > > ----- ERROR ---------------------------- >>>> > > > > >>>> > > > > SQL Error: RESOURCE ERROR: Waited for 30000 ms, but only 10 >>>> tasks for >>>> > > > > 'Fetch parquet metadata' are complete. Total number of tasks 29, >>>> > > > > parallelism 16. >>>> > > > > >>>> > > > > >>>> > > > > [Error Id: 3b079174-f5d0-4313-8097-25a0b3070854 on >>>> > > > > IORA-G9KY9P2.stf.nus.edu.sg:31010] >>>> > > > > RESOURCE ERROR: Waited for 30000 ms, but only 10 tasks for >>>> 'Fetch >>>> > > > parquet >>>> > > > > metadata' are complete. Total number of tasks 29, parallelism >>>> 16. >>>> > > > > >>>> > > > > >>>> > > > > [Error Id: 3b079174-f5d0-4313-8097-25a0b3070854 on >>>> > > > > IORA-G9KY9P2.stf.nus.edu.sg:31010] >>>> > > > > RESOURCE ERROR: Waited for 30000 ms, but only 10 tasks for >>>> 'Fetch >>>> > > > > parquet metadata' are complete. Total number of tasks 29, >>>> parallelism >>>> > > 16. >>>> > > > > >>>> > > > > >>>> > > > > [Error Id: 3b079174-f5d0-4313-8097-25a0b3070854 on >>>> > > > > IORA-G9KY9P2.stf.nus.edu.sg:31010] >>>> > > > > RESOURCE ERROR: Waited for 30000 ms, but only 10 tasks for >>>> 'Fetch >>>> > > > > parquet metadata' are complete. Total number of tasks 29, >>>> parallelism >>>> > > 16. >>>> > > > > >>>> > > > > >>>> > > > > [Error Id: 3b079174-f5d0-4313-8097-25a0b3070854 on >>>> > > > > IORA-G9KY9P2.stf.nus.edu.sg:31010] >>>> > > > > >>>> > > > > ---------------------------------------- >>>> > > > > From Drill log: >>>> > > > > >>>> > > > > 2018-09-20 08:58:12,035 >>>> > [245d0f5a-ae5f-bfa2-ff04-40f7bdd1c2bf:foreman] >>>> > > > > INFO o.a.drill.exec.work.foreman.Foreman - Query text for query >>>> id >>>> > > > > 245d0f5a-ae5f-bfa2-ff04-40f7bdd1c2bf: select * >>>> > > > > from dfs.tmp.load_pos_sales_detail_tbl >>>> > > > > >>>> > > > > 2018-09-20 08:58:53,068 >>>> > [245d0f5a-ae5f-bfa2-ff04-40f7bdd1c2bf:foreman] >>>> > > > > ERROR o.a.d.e.s.parquet.metadata.Metadata - Waited for 30000 >>>> ms, but >>>> > > > only >>>> > > > > 10 tasks for 'Fetch parquet metadata' are complete. Total >>>> number of >>>> > > tasks >>>> > > > > 29, parallelism 16. >>>> > > > > java.util.concurrent.CancellationException: null >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.TimedCallable$FutureMapper. >>>> > > > apply(TimedCallable.java:86) >>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.TimedCallable$FutureMapper. >>>> > > > apply(TimedCallable.java:57) >>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.common.collections.Collectors.lambda$ >>>> > > > toList$2(Collectors.java:97) >>>> > > > > ~[drill-common-1.14.0.jar:1.14.0] >>>> > > > > at java.util.ArrayList.forEach(ArrayList.java:1257) >>>> ~[na:1.8.0_172] >>>> > > > > at >>>> > > > > org.apache.drill.common.collections.Collectors.toList( >>>> > > > Collectors.java:97) >>>> > > > > ~[drill-common-1.14.0.jar:1.14.0] >>>> > > > > at org.apache.drill.exec.store.TimedCallable.run( >>>> > > TimedCallable.java:214) >>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata. >>>> > > > getParquetFileMetadata_v3(Metadata.java:340) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata. >>>> > > > getParquetTableMetadata(Metadata.java:324) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata. >>>> > > > getParquetTableMetadata(Metadata.java:305) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata. >>>> > > > getParquetTableMetadata(Metadata.java:124) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan. >>>> > > > initInternal(ParquetGroupScan.java:254) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.parquet.AbstractParquetGroupScan.init( >>>> > > > AbstractParquetGroupScan.java:380) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan. >>>> > > > init>(ParquetGroupScan.java:132) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan. >>>> > > > init>(ParquetGroupScan.java:102) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan( >>>> > > > ParquetFormatPlugin.java:180) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan( >>>> > > > ParquetFormatPlugin.java:70) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan( >>>> > > > FileSystemPlugin.java:136) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan( >>>> > > > AbstractStoragePlugin.java:116) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan( >>>> > > > AbstractStoragePlugin.java:111) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillTable. >>>> > > > getGroupScan(DrillTable.java:99) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.( >>>> > > > DrillScanRel.java:89) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.( >>>> > > > DrillScanRel.java:69) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.( >>>> > > > DrillScanRel.java:62) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRule.onMatch( >>>> > > > DrillScanRule.java:38) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.calcite.plan.volcano.VolcanoRuleCall. >>>> > > > onMatch(VolcanoRuleCall.java:212) >>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.calcite.plan.volcano.VolcanoPlanner. >>>> > > > findBestExp(VolcanoPlanner.java:652) >>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6] >>>> > > > > at org.apache.calcite.tools.Programs$RuleSetProgram.run( >>>> > > > Programs.java:368) >>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.handlers. >>>> > > DefaultSqlHandler.transform( >>>> > > > DefaultSqlHandler.java:429) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.handlers. >>>> > > DefaultSqlHandler.transform( >>>> > > > DefaultSqlHandler.java:369) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler. >>>> > > > convertToRawDrel(DefaultSqlHandler.java:255) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler. >>>> > > > convertToDrel(DefaultSqlHandler.java:318) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan( >>>> > > > DefaultSqlHandler.java:180) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.DrillSqlWorker. >>>> > > > getQueryPlan(DrillSqlWorker.java:145) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan( >>>> > > > DrillSqlWorker.java:83) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at org.apache.drill.exec.work >>>> > .foreman.Foreman.runSQL(Foreman.java:567) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at org.apache.drill.exec.work >>>> .foreman.Foreman.run(Foreman.java:266) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> > > > ThreadPoolExecutor.java:1149) >>>> > > > > [na:1.8.0_172] >>>> > > > > at >>>> > > > > >>>> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> > > > ThreadPoolExecutor.java:624) >>>> > > > > [na:1.8.0_172] >>>> > > > > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_172] >>>> > > > > 2018-09-20 08:58:53,080 >>>> > [245d0f5a-ae5f-bfa2-ff04-40f7bdd1c2bf:foreman] >>>> > > > > INFO o.a.d.e.s.parquet.metadata.Metadata - User Error Occurred: >>>> > Waited >>>> > > > for >>>> > > > > 30000 ms, but only 10 tasks for 'Fetch parquet metadata' are >>>> > complete. >>>> > > > > Total number of tasks 29, parallelism 16. (null) >>>> > > > > org.apache.drill.common.exceptions.UserException: RESOURCE >>>> ERROR: >>>> > > Waited >>>> > > > > for 30000 ms, but only 10 tasks for 'Fetch parquet metadata' are >>>> > > > complete. >>>> > > > > Total number of tasks 29, parallelism 16. >>>> > > > > >>>> > > > > >>>> > > > > [Error Id: f887dcae-9f55-469c-be52-b6ce2a37eeb0 ] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.common.exceptions.UserException$ >>>> > > > Builder.build(UserException.java:633) >>>> > > > > ~[drill-common-1.14.0.jar:1.14.0] >>>> > > > > at org.apache.drill.exec.store.TimedCallable.run( >>>> > > TimedCallable.java:253) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata. >>>> > > > getParquetFileMetadata_v3(Metadata.java:340) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata. >>>> > > > getParquetTableMetadata(Metadata.java:324) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata. >>>> > > > getParquetTableMetadata(Metadata.java:305) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata. >>>> > > > getParquetTableMetadata(Metadata.java:124) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan. >>>> > > > initInternal(ParquetGroupScan.java:254) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.parquet.AbstractParquetGroupScan.init( >>>> > > > AbstractParquetGroupScan.java:380) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan. >>>> > > > init>(ParquetGroupScan.java:132) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan. >>>> > > > init>(ParquetGroupScan.java:102) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan( >>>> > > > ParquetFormatPlugin.java:180) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan( >>>> > > > ParquetFormatPlugin.java:70) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan( >>>> > > > FileSystemPlugin.java:136) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan( >>>> > > > AbstractStoragePlugin.java:116) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan( >>>> > > > AbstractStoragePlugin.java:111) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillTable. >>>> > > > getGroupScan(DrillTable.java:99) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.( >>>> > > > DrillScanRel.java:89) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.( >>>> > > > DrillScanRel.java:69) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.( >>>> > > > DrillScanRel.java:62) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRule.onMatch( >>>> > > > DrillScanRule.java:38) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.calcite.plan.volcano.VolcanoRuleCall. >>>> > > > onMatch(VolcanoRuleCall.java:212) >>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.calcite.plan.volcano.VolcanoPlanner. >>>> > > > findBestExp(VolcanoPlanner.java:652) >>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6] >>>> > > > > at org.apache.calcite.tools.Programs$RuleSetProgram.run( >>>> > > > Programs.java:368) >>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.handlers. >>>> > > DefaultSqlHandler.transform( >>>> > > > DefaultSqlHandler.java:429) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.handlers. >>>> > > DefaultSqlHandler.transform( >>>> > > > DefaultSqlHandler.java:369) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler. >>>> > > > convertToRawDrel(DefaultSqlHandler.java:255) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler. >>>> > > > convertToDrel(DefaultSqlHandler.java:318) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan( >>>> > > > DefaultSqlHandler.java:180) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.DrillSqlWorker. >>>> > > > getQueryPlan(DrillSqlWorker.java:145) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan( >>>> > > > DrillSqlWorker.java:83) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at org.apache.drill.exec.work >>>> > .foreman.Foreman.runSQL(Foreman.java:567) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at org.apache.drill.exec.work >>>> .foreman.Foreman.run(Foreman.java:266) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> > > > ThreadPoolExecutor.java:1149) >>>> > > > > [na:1.8.0_172] >>>> > > > > at >>>> > > > > >>>> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> > > > ThreadPoolExecutor.java:624) >>>> > > > > [na:1.8.0_172] >>>> > > > > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_172] >>>> > > > > Caused by: java.util.concurrent.CancellationException: null >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.TimedCallable$FutureMapper. >>>> > > > apply(TimedCallable.java:86) >>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.TimedCallable$FutureMapper. >>>> > > > apply(TimedCallable.java:57) >>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.common.collections.Collectors.lambda$ >>>> > > > toList$2(Collectors.java:97) >>>> > > > > ~[drill-common-1.14.0.jar:1.14.0] >>>> > > > > at java.util.ArrayList.forEach(ArrayList.java:1257) >>>> ~[na:1.8.0_172] >>>> > > > > at >>>> > > > > org.apache.drill.common.collections.Collectors.toList( >>>> > > > Collectors.java:97) >>>> > > > > ~[drill-common-1.14.0.jar:1.14.0] >>>> > > > > at org.apache.drill.exec.store.TimedCallable.run( >>>> > > TimedCallable.java:214) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > ... 33 common frames omitted >>>> > > > > 2018-09-20 09:02:10,608 [UserServer-1] WARN >>>> > > > > o.a.drill.exec.rpc.user.UserServer - Message of mode REQUEST of >>>> rpc >>>> > > > type 3 >>>> > > > > took longer than 500ms. Actual duration was 2042ms. >>>> > > > > 2018-09-20 09:02:10,608 >>>> > [245d0e6f-0dc1-2a4b-12a4-b9aaad4182fc:foreman] >>>> > > > > INFO o.a.drill.exec.work.foreman.Foreman - Query text for query >>>> id >>>> > > > > 245d0e6f-0dc1-2a4b-12a4-b9aaad4182fc: select * >>>> > > > > from dfs.tmp.load_pos_sales_detail_tbl >>>> > > > > >>>> > > > > 2018-09-20 09:02:42,615 >>>> > [245d0e6f-0dc1-2a4b-12a4-b9aaad4182fc:foreman] >>>> > > > > ERROR o.a.d.e.s.parquet.metadata.Metadata - Waited for 30000 >>>> ms, but >>>> > > > only >>>> > > > > 10 tasks for 'Fetch parquet metadata' are complete. Total >>>> number of >>>> > > tasks >>>> > > > > 29, parallelism 16. >>>> > > > > java.util.concurrent.CancellationException: null >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.TimedCallable$FutureMapper. >>>> > > > apply(TimedCallable.java:86) >>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.TimedCallable$FutureMapper. >>>> > > > apply(TimedCallable.java:57) >>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.common.collections.Collectors.lambda$ >>>> > > > toList$2(Collectors.java:97) >>>> > > > > ~[drill-common-1.14.0.jar:1.14.0] >>>> > > > > at java.util.ArrayList.forEach(ArrayList.java:1257) >>>> ~[na:1.8.0_172] >>>> > > > > at >>>> > > > > org.apache.drill.common.collections.Collectors.toList( >>>> > > > Collectors.java:97) >>>> > > > > ~[drill-common-1.14.0.jar:1.14.0] >>>> > > > > at org.apache.drill.exec.store.TimedCallable.run( >>>> > > TimedCallable.java:214) >>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata. >>>> > > > getParquetFileMetadata_v3(Metadata.java:340) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata. >>>> > > > getParquetTableMetadata(Metadata.java:324) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata. >>>> > > > getParquetTableMetadata(Metadata.java:305) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata. >>>> > > > getParquetTableMetadata(Metadata.java:124) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan. >>>> > > > initInternal(ParquetGroupScan.java:254) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.parquet.AbstractParquetGroupScan.init( >>>> > > > AbstractParquetGroupScan.java:380) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan. >>>> > > > init>(ParquetGroupScan.java:132) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan. >>>> > > > init>(ParquetGroupScan.java:102) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan( >>>> > > > ParquetFormatPlugin.java:180) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan( >>>> > > > ParquetFormatPlugin.java:70) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan( >>>> > > > FileSystemPlugin.java:136) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan( >>>> > > > AbstractStoragePlugin.java:116) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan( >>>> > > > AbstractStoragePlugin.java:111) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillTable. >>>> > > > getGroupScan(DrillTable.java:99) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.( >>>> > > > DrillScanRel.java:89) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.( >>>> > > > DrillScanRel.java:69) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.( >>>> > > > DrillScanRel.java:62) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRule.onMatch( >>>> > > > DrillScanRule.java:38) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.calcite.plan.volcano.VolcanoRuleCall. >>>> > > > onMatch(VolcanoRuleCall.java:212) >>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.calcite.plan.volcano.VolcanoPlanner. >>>> > > > findBestExp(VolcanoPlanner.java:652) >>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6] >>>> > > > > at org.apache.calcite.tools.Programs$RuleSetProgram.run( >>>> > > > Programs.java:368) >>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.handlers. >>>> > > DefaultSqlHandler.transform( >>>> > > > DefaultSqlHandler.java:429) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.handlers. >>>> > > DefaultSqlHandler.transform( >>>> > > > DefaultSqlHandler.java:369) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler. >>>> > > > convertToRawDrel(DefaultSqlHandler.java:255) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler. >>>> > > > convertToDrel(DefaultSqlHandler.java:318) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan( >>>> > > > DefaultSqlHandler.java:180) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.DrillSqlWorker. >>>> > > > getQueryPlan(DrillSqlWorker.java:145) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan( >>>> > > > DrillSqlWorker.java:83) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at org.apache.drill.exec.work >>>> > .foreman.Foreman.runSQL(Foreman.java:567) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at org.apache.drill.exec.work >>>> .foreman.Foreman.run(Foreman.java:266) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> > > > ThreadPoolExecutor.java:1149) >>>> > > > > [na:1.8.0_172] >>>> > > > > at >>>> > > > > >>>> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> > > > ThreadPoolExecutor.java:624) >>>> > > > > [na:1.8.0_172] >>>> > > > > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_172] >>>> > > > > 2018-09-20 09:02:42,625 >>>> > [245d0e6f-0dc1-2a4b-12a4-b9aaad4182fc:foreman] >>>> > > > > INFO o.a.d.e.s.parquet.metadata.Metadata - User Error Occurred: >>>> > Waited >>>> > > > for >>>> > > > > 30000 ms, but only 10 tasks for 'Fetch parquet metadata' are >>>> > complete. >>>> > > > > Total number of tasks 29, parallelism 16. (null) >>>> > > > > org.apache.drill.common.exceptions.UserException: RESOURCE >>>> ERROR: >>>> > > Waited >>>> > > > > for 30000 ms, but only 10 tasks for 'Fetch parquet metadata' are >>>> > > > complete. >>>> > > > > Total number of tasks 29, parallelism 16. >>>> > > > > >>>> > > > > >>>> > > > > [Error Id: 3b079174-f5d0-4313-8097-25a0b3070854 ] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.common.exceptions.UserException$ >>>> > > > Builder.build(UserException.java:633) >>>> > > > > ~[drill-common-1.14.0.jar:1.14.0] >>>> > > > > at org.apache.drill.exec.store.TimedCallable.run( >>>> > > TimedCallable.java:253) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata. >>>> > > > getParquetFileMetadata_v3(Metadata.java:340) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata. >>>> > > > getParquetTableMetadata(Metadata.java:324) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata. >>>> > > > getParquetTableMetadata(Metadata.java:305) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.metadata.Metadata. >>>> > > > getParquetTableMetadata(Metadata.java:124) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan. >>>> > > > initInternal(ParquetGroupScan.java:254) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.parquet.AbstractParquetGroupScan.init( >>>> > > > AbstractParquetGroupScan.java:380) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan. >>>> > > > init>(ParquetGroupScan.java:132) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.parquet.ParquetGroupScan. >>>> > > > init>(ParquetGroupScan.java:102) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan( >>>> > > > ParquetFormatPlugin.java:180) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.parquet.ParquetFormatPlugin.getGroupScan( >>>> > > > ParquetFormatPlugin.java:70) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.dfs.FileSystemPlugin.getPhysicalScan( >>>> > > > FileSystemPlugin.java:136) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan( >>>> > > > AbstractStoragePlugin.java:116) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan( >>>> > > > AbstractStoragePlugin.java:111) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillTable. >>>> > > > getGroupScan(DrillTable.java:99) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.( >>>> > > > DrillScanRel.java:89) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.( >>>> > > > DrillScanRel.java:69) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRel.( >>>> > > > DrillScanRel.java:62) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.logical.DrillScanRule.onMatch( >>>> > > > DrillScanRule.java:38) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.calcite.plan.volcano.VolcanoRuleCall. >>>> > > > onMatch(VolcanoRuleCall.java:212) >>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.calcite.plan.volcano.VolcanoPlanner. >>>> > > > findBestExp(VolcanoPlanner.java:652) >>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6] >>>> > > > > at org.apache.calcite.tools.Programs$RuleSetProgram.run( >>>> > > > Programs.java:368) >>>> > > > > [calcite-core-1.16.0-drill-r6.jar:1.16.0-drill-r6] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.handlers. >>>> > > DefaultSqlHandler.transform( >>>> > > > DefaultSqlHandler.java:429) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.handlers. >>>> > > DefaultSqlHandler.transform( >>>> > > > DefaultSqlHandler.java:369) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler. >>>> > > > convertToRawDrel(DefaultSqlHandler.java:255) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler. >>>> > > > convertToDrel(DefaultSqlHandler.java:318) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > >>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan( >>>> > > > DefaultSqlHandler.java:180) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.DrillSqlWorker. >>>> > > > getQueryPlan(DrillSqlWorker.java:145) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan( >>>> > > > DrillSqlWorker.java:83) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at org.apache.drill.exec.work >>>> > .foreman.Foreman.runSQL(Foreman.java:567) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at org.apache.drill.exec.work >>>> .foreman.Foreman.run(Foreman.java:266) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> > > > ThreadPoolExecutor.java:1149) >>>> > > > > [na:1.8.0_172] >>>> > > > > at >>>> > > > > >>>> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> > > > ThreadPoolExecutor.java:624) >>>> > > > > [na:1.8.0_172] >>>> > > > > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_172] >>>> > > > > Caused by: java.util.concurrent.CancellationException: null >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.TimedCallable$FutureMapper. >>>> > > > apply(TimedCallable.java:86) >>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.exec.store.TimedCallable$FutureMapper. >>>> > > > apply(TimedCallable.java:57) >>>> > > > > ~[drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > at >>>> > > > > >>>> > > > > org.apache.drill.common.collections.Collectors.lambda$ >>>> > > > toList$2(Collectors.java:97) >>>> > > > > ~[drill-common-1.14.0.jar:1.14.0] >>>> > > > > at java.util.ArrayList.forEach(ArrayList.java:1257) >>>> ~[na:1.8.0_172] >>>> > > > > at >>>> > > > > org.apache.drill.common.collections.Collectors.toList( >>>> > > > Collectors.java:97) >>>> > > > > ~[drill-common-1.14.0.jar:1.14.0] >>>> > > > > at org.apache.drill.exec.store.TimedCallable.run( >>>> > > TimedCallable.java:214) >>>> > > > > [drill-java-exec-1.14.0.jar:1.14.0] >>>> > > > > ... 33 common frames omitted >>>> > > > > >>>> > > > > >>>> > > > > ---------------------------------------- >>>> > > > > ---------- >>>> > > > > >>>> > > > >>>> > > >>>> > >>>> >>>
