[jira] [Commented] (DRILL-3721) Regarding drill with big file
[ https://issues.apache.org/jira/browse/DRILL-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723118#comment-14723118 ] kunal commented on DRILL-3721: -- [~aengelbrecht] I have attached the sys.options properties > Regarding drill with big file > - > > Key: DRILL-3721 > URL: https://issues.apache.org/jira/browse/DRILL-3721 > Project: Apache Drill > Issue Type: Bug >Reporter: kunal > Attachments: sample.json, sqlline.log, sys_Options.png, > sys_Options_1.png > > > I am new to apache drill. I have configured apache drill on machine with > centos. > "DRILL_MAX_DIRECT_MEMORY" = 25g > "DRILL_HEAP" = 4g > I have a 600 mb and 3 gb json file [sample file attached]. When i fire query > on relativly small size file everything works fine but as I fire same query > with 600 mb and 3 gb files it gives following error[stack trace attached]. > Query - > select tbl5.product_id product_id,tbl5.gender gender,tbl5.item_number > item_number,tbl5.price price,tbl5.description > description,tbl5.color_swatch.image image,tbl5.color_swatch.color color from > (select tbl4.product_id product_id,tbl4.gender gender,tbl4.item_number > item_number,tbl4.price price,tbl4.size.description > description,FLATTEN(tbl4.size.color_swatch) color_swatch from > (select tbl3.product_id product_id,tbl3.catalog_item.gender > gender,tbl3.catalog_item.item_number item_number,tbl3.catalog_item.price > price,FLATTEN(tbl3.catalog_item.size) size from > (select tbl2.product.product_id as > product_id,FLATTEN(tbl2.product.catalog_item) as catalog_item from > (select FLATTEN(tbl1.catalog.product) product from dfs.root.`demo.json` tbl1) > tbl2) tbl3) tbl4) tbl5 > -- > Error - > SYSTEM ERROR: IllegalArgumentException: initialCapacity: -2147483648 > (expectd: 0+) > Fragment 0:0 > [Error Id: 60cf1b95-762d-4a0d-8cae-a2db418d4ea9 on sinhagad:31010] > -- > 1) Am i doing someting wrong or missing something ( probably because i am not > using cluster ?? ). > Please guide me through this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3721) Regarding drill with big file
[ https://issues.apache.org/jira/browse/DRILL-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14720005#comment-14720005 ] Abhishek Girish commented on DRILL-3721: Looks similar to DRILL-2882. If confirmed, this issue can be marked duplicate, and the earlier issue prioritized. > Regarding drill with big file > - > > Key: DRILL-3721 > URL: https://issues.apache.org/jira/browse/DRILL-3721 > Project: Apache Drill > Issue Type: Bug >Reporter: kunal > Attachments: sample.json, sqlline.log > > > I am new to apache drill. I have configured apache drill on machine with > centos. > "DRILL_MAX_DIRECT_MEMORY" = 25g > "DRILL_HEAP" = 4g > I have a 600 mb and 3 gb json file [sample file attached]. When i fire query > on relativly small size file everything works fine but as I fire same query > with 600 mb and 3 gb files it gives following error[stack trace attached]. > Query - > select tbl5.product_id product_id,tbl5.gender gender,tbl5.item_number > item_number,tbl5.price price,tbl5.description > description,tbl5.color_swatch.image image,tbl5.color_swatch.color color from > (select tbl4.product_id product_id,tbl4.gender gender,tbl4.item_number > item_number,tbl4.price price,tbl4.size.description > description,FLATTEN(tbl4.size.color_swatch) color_swatch from > (select tbl3.product_id product_id,tbl3.catalog_item.gender > gender,tbl3.catalog_item.item_number item_number,tbl3.catalog_item.price > price,FLATTEN(tbl3.catalog_item.size) size from > (select tbl2.product.product_id as > product_id,FLATTEN(tbl2.product.catalog_item) as catalog_item from > (select FLATTEN(tbl1.catalog.product) product from dfs.root.`demo.json` tbl1) > tbl2) tbl3) tbl4) tbl5 > -- > Error - > SYSTEM ERROR: IllegalArgumentException: initialCapacity: -2147483648 > (expectd: 0+) > Fragment 0:0 > [Error Id: 60cf1b95-762d-4a0d-8cae-a2db418d4ea9 on sinhagad:31010] > -- > 1) Am i doing someting wrong or missing something ( probably because i am not > using cluster ?? ). > Please guide me through this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3721) Regarding drill with big file
[ https://issues.apache.org/jira/browse/DRILL-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14719296#comment-14719296 ] Andries Engelbrecht commented on DRILL-3721: See what the query memory per node is set at and increase it to see if it resolves your problem. The parameter is planner.memory.max_query_memory_per_node Query sys.options to see what it is set as and use alter system to modify. https://drill.apache.org/docs/configuring-drill-memory/ https://drill.apache.org/docs/alter-system/ https://drill.apache.org/docs/configuration-options-introduction/ > Regarding drill with big file > - > > Key: DRILL-3721 > URL: https://issues.apache.org/jira/browse/DRILL-3721 > Project: Apache Drill > Issue Type: Bug >Reporter: kunal > Attachments: sample.json, sqlline.log > > > I am new to apache drill. I have configured apache drill on machine with > centos. > "DRILL_MAX_DIRECT_MEMORY" = 25g > "DRILL_HEAP" = 4g > I have a 600 mb and 3 gb json file [sample file attached]. When i fire query > on relativly small size file everything works fine but as I fire same query > with 600 mb and 3 gb files it gives following error[stack trace attached]. > Query - > select tbl5.product_id product_id,tbl5.gender gender,tbl5.item_number > item_number,tbl5.price price,tbl5.description > description,tbl5.color_swatch.image image,tbl5.color_swatch.color color from > (select tbl4.product_id product_id,tbl4.gender gender,tbl4.item_number > item_number,tbl4.price price,tbl4.size.description > description,FLATTEN(tbl4.size.color_swatch) color_swatch from > (select tbl3.product_id product_id,tbl3.catalog_item.gender > gender,tbl3.catalog_item.item_number item_number,tbl3.catalog_item.price > price,FLATTEN(tbl3.catalog_item.size) size from > (select tbl2.product.product_id as > product_id,FLATTEN(tbl2.product.catalog_item) as catalog_item from > (select FLATTEN(tbl1.catalog.product) product from dfs.root.`demo.json` tbl1) > tbl2) tbl3) tbl4) tbl5 > -- > Error - > SYSTEM ERROR: IllegalArgumentException: initialCapacity: -2147483648 > (expectd: 0+) > Fragment 0:0 > [Error Id: 60cf1b95-762d-4a0d-8cae-a2db418d4ea9 on sinhagad:31010] > -- > 1) Am i doing someting wrong or missing something ( probably because i am not > using cluster ?? ). > Please guide me through this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
