[jira] [Resolved] (DRILL-4208) Storage plugin configuration persistence not working for Apache Drill
[
https://issues.apache.org/jira/browse/DRILL-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristine Hahn resolved DRILL-4208.
--
Resolution: Fixed
Fix Version/s: (was: Future)
1.5.0
Fixed doc per info from Steven:
http://apache.github.io/drill/docs/storage-plugin-registration/#storage-plugin-configuration-persistence
Thanks for raising this issue, Devender.
> Storage plugin configuration persistence not working for Apache Drill
> -
>
> Key: DRILL-4208
> URL: https://issues.apache.org/jira/browse/DRILL-4208
> Project: Apache Drill
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.3.0
> Environment: Ubuntu 14.0.4
>Reporter: Devender Yadav
>Assignee: Kristine Hahn
> Fix For: 1.5.0
>
>
> According to Drill's documentation :
> Drill uses /tmp/drill/sys.storage_plugins to store storage plugin
> configurations. The temporary directory clears when you quit the Drill shell.
> To save your storage plugin configurations from one session to the next, set
> the following option in the drill-override.conf file if you are running Drill
> in embedded mode.
> drill.exec.sys.store.provider.local.path = "/mypath"
> I checked /tmp/drill/sys.storage_plugins, there is some data in this file.
> Then I modified drill-override.conf :
> drill.exec: {
> cluster-id: "drillbits1",
> zk.connect: "localhost:2181",
> drill.exec.sys.store.provider.local.path = "/home/dev/abc"
> }
> I restarted drill & even restarted my machine. Nothing is created at this
> location.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (DRILL-4208) Storage plugin configuration persistence not working for Apache Drill
[
https://issues.apache.org/jira/browse/DRILL-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristine Hahn reassigned DRILL-4208:
Assignee: Kristine Hahn
> Storage plugin configuration persistence not working for Apache Drill
> -
>
> Key: DRILL-4208
> URL: https://issues.apache.org/jira/browse/DRILL-4208
> Project: Apache Drill
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.3.0
> Environment: Ubuntu 14.0.4
>Reporter: Devender Yadav
>Assignee: Kristine Hahn
> Fix For: Future
>
>
> According to Drill's documentation :
> Drill uses /tmp/drill/sys.storage_plugins to store storage plugin
> configurations. The temporary directory clears when you quit the Drill shell.
> To save your storage plugin configurations from one session to the next, set
> the following option in the drill-override.conf file if you are running Drill
> in embedded mode.
> drill.exec.sys.store.provider.local.path = "/mypath"
> I checked /tmp/drill/sys.storage_plugins, there is some data in this file.
> Then I modified drill-override.conf :
> drill.exec: {
> cluster-id: "drillbits1",
> zk.connect: "localhost:2181",
> drill.exec.sys.store.provider.local.path = "/home/dev/abc"
> }
> I restarted drill & even restarted my machine. Nothing is created at this
> location.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-4208) Storage plugin configuration persistence not working for Apache Drill
[
https://issues.apache.org/jira/browse/DRILL-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristine Hahn updated DRILL-4208:
-
Component/s: Documentation
> Storage plugin configuration persistence not working for Apache Drill
> -
>
> Key: DRILL-4208
> URL: https://issues.apache.org/jira/browse/DRILL-4208
> Project: Apache Drill
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.3.0
> Environment: Ubuntu 14.0.4
>Reporter: Devender Yadav
> Fix For: Future
>
>
> According to Drill's documentation :
> Drill uses /tmp/drill/sys.storage_plugins to store storage plugin
> configurations. The temporary directory clears when you quit the Drill shell.
> To save your storage plugin configurations from one session to the next, set
> the following option in the drill-override.conf file if you are running Drill
> in embedded mode.
> drill.exec.sys.store.provider.local.path = "/mypath"
> I checked /tmp/drill/sys.storage_plugins, there is some data in this file.
> Then I modified drill-override.conf :
> drill.exec: {
> cluster-id: "drillbits1",
> zk.connect: "localhost:2181",
> drill.exec.sys.store.provider.local.path = "/home/dev/abc"
> }
> I restarted drill & even restarted my machine. Nothing is created at this
> location.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (DRILL-3985) 'configuring-web-ui-and-rest-api-security' missing from navigation bar
[ https://issues.apache.org/jira/browse/DRILL-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kristine Hahn resolved DRILL-3985. -- Resolution: Fixed Resolved by DRILL-4080 > 'configuring-web-ui-and-rest-api-security' missing from navigation bar > -- > > Key: DRILL-3985 > URL: https://issues.apache.org/jira/browse/DRILL-3985 > Project: Apache Drill > Issue Type: Bug > Components: Documentation > Environment: n/a >Reporter: Andy Pernsteiner >Assignee: Kristine Hahn > > Using the search box on the doc page, I was able to find the following page : > https://drill.apache.org/docs/configuring-web-ui-and-rest-api-security/ > Once I navigate directly to that link, I notice that the navigation-bar to > the left shows configuring-web-ui-and-rest-api-security . All is well. > However, If I open another browser tab, and simply go to the > https://drill.apache.org/docs/ url (top level), then click on the 'Configure > Drill' link (under which the earlier mentioned page was listed), I do NOT see > configuring-web-ui-and-rest-api-security . > Instead, I see configuring-web-console-and-rest-api-security (notice the > word 'console'). Further (worse) is that the content is vastly different. > I'm not sure if these two pages were supposed to *BOTH* exist..or if one was > a draft, and the 'hidden' one is what we want to keep. > In any case, this will lead to confusion if we don't fix the nav-bar and make > sure the correct pages are accessible directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (DRILL-3985) 'configuring-web-ui-and-rest-api-security' missing from navigation bar
[ https://issues.apache.org/jira/browse/DRILL-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kristine Hahn reassigned DRILL-3985: Assignee: Kristine Hahn (was: Bridget Bevens) > 'configuring-web-ui-and-rest-api-security' missing from navigation bar > -- > > Key: DRILL-3985 > URL: https://issues.apache.org/jira/browse/DRILL-3985 > Project: Apache Drill > Issue Type: Bug > Components: Documentation > Environment: n/a >Reporter: Andy Pernsteiner >Assignee: Kristine Hahn > > Using the search box on the doc page, I was able to find the following page : > https://drill.apache.org/docs/configuring-web-ui-and-rest-api-security/ > Once I navigate directly to that link, I notice that the navigation-bar to > the left shows configuring-web-ui-and-rest-api-security . All is well. > However, If I open another browser tab, and simply go to the > https://drill.apache.org/docs/ url (top level), then click on the 'Configure > Drill' link (under which the earlier mentioned page was listed), I do NOT see > configuring-web-ui-and-rest-api-security . > Instead, I see configuring-web-console-and-rest-api-security (notice the > word 'console'). Further (worse) is that the content is vastly different. > I'm not sure if these two pages were supposed to *BOTH* exist..or if one was > a draft, and the 'hidden' one is what we want to keep. > In any case, this will lead to confusion if we don't fix the nav-bar and make > sure the correct pages are accessible directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DRILL-4205) Simple query hit IndexOutOfBoundException
[
https://issues.apache.org/jira/browse/DRILL-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Parth Chandra resolved DRILL-4205.
--
Resolution: Fixed
> Simple query hit IndexOutOfBoundException
> --
>
> Key: DRILL-4205
> URL: https://issues.apache.org/jira/browse/DRILL-4205
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Dechang Gu
>Assignee: Parth Chandra
>
> The following query failed due to IOB:
> 0: jdbc:drill:schema=wf_pigprq100> select * from
> `store_sales/part-m-00073.parquet`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: srcIndex: 1048587
> Fragment 0:0
> [Error Id: ad8d2bc0-259f-483c-9024-93865963541e on ucs-node4.perf.lab:31010]
> (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet
> record reader.
> Message:
> Hadoop path: /tpcdsPigParq/SF100/store_sales/part-m-00073.parquet
> Total records read: 135280
> Mock records read: 0
> Records to read: 1424
> Row group index: 0
> Records in row group: 3775712
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message pig_schema {
> optional int64 ss_sold_date_sk;
> optional int64 ss_sold_time_sk;
> optional int64 ss_item_sk;
> optional int64 ss_customer_sk;
> optional int64 ss_cdemo_sk;
> optional int64 ss_hdemo_sk;
> optional int64 ss_addr_sk;
> optional int64 ss_store_sk;
> optional int64 ss_promo_sk;
> optional int64 ss_ticket_number;
> optional int64 ss_quantity;
> optional double ss_wholesale_cost;
> optional double ss_list_price;
> optional double ss_sales_price;
> optional double ss_ext_discount_amt;
> optional double ss_ext_sales_price;
> optional double ss_ext_wholesale_cost;
> optional double ss_ext_list_price;
> optional double ss_ext_tax;
> optional double ss_coupon_amt;
> optional double ss_net_paid;
> optional double ss_net_paid_inc_tax;
> optional double ss_net_profit;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4205) Simple query hit IndexOutOfBoundException
[
https://issues.apache.org/jira/browse/DRILL-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063274#comment-15063274
]
ASF GitHub Bot commented on DRILL-4205:
---
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/306
> Simple query hit IndexOutOfBoundException
> --
>
> Key: DRILL-4205
> URL: https://issues.apache.org/jira/browse/DRILL-4205
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Dechang Gu
>Assignee: Parth Chandra
>
> The following query failed due to IOB:
> 0: jdbc:drill:schema=wf_pigprq100> select * from
> `store_sales/part-m-00073.parquet`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: srcIndex: 1048587
> Fragment 0:0
> [Error Id: ad8d2bc0-259f-483c-9024-93865963541e on ucs-node4.perf.lab:31010]
> (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet
> record reader.
> Message:
> Hadoop path: /tpcdsPigParq/SF100/store_sales/part-m-00073.parquet
> Total records read: 135280
> Mock records read: 0
> Records to read: 1424
> Row group index: 0
> Records in row group: 3775712
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message pig_schema {
> optional int64 ss_sold_date_sk;
> optional int64 ss_sold_time_sk;
> optional int64 ss_item_sk;
> optional int64 ss_customer_sk;
> optional int64 ss_cdemo_sk;
> optional int64 ss_hdemo_sk;
> optional int64 ss_addr_sk;
> optional int64 ss_store_sk;
> optional int64 ss_promo_sk;
> optional int64 ss_ticket_number;
> optional int64 ss_quantity;
> optional double ss_wholesale_cost;
> optional double ss_list_price;
> optional double ss_sales_price;
> optional double ss_ext_discount_amt;
> optional double ss_ext_sales_price;
> optional double ss_ext_wholesale_cost;
> optional double ss_ext_list_price;
> optional double ss_ext_tax;
> optional double ss_coupon_amt;
> optional double ss_net_paid;
> optional double ss_net_paid_inc_tax;
> optional double ss_net_profit;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4205) Simple query hit IndexOutOfBoundException
[
https://issues.apache.org/jira/browse/DRILL-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063277#comment-15063277
]
Parth Chandra commented on DRILL-4205:
--
Fixed in abdfc6a
> Simple query hit IndexOutOfBoundException
> --
>
> Key: DRILL-4205
> URL: https://issues.apache.org/jira/browse/DRILL-4205
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Dechang Gu
>Assignee: Parth Chandra
>
> The following query failed due to IOB:
> 0: jdbc:drill:schema=wf_pigprq100> select * from
> `store_sales/part-m-00073.parquet`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: srcIndex: 1048587
> Fragment 0:0
> [Error Id: ad8d2bc0-259f-483c-9024-93865963541e on ucs-node4.perf.lab:31010]
> (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet
> record reader.
> Message:
> Hadoop path: /tpcdsPigParq/SF100/store_sales/part-m-00073.parquet
> Total records read: 135280
> Mock records read: 0
> Records to read: 1424
> Row group index: 0
> Records in row group: 3775712
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message pig_schema {
> optional int64 ss_sold_date_sk;
> optional int64 ss_sold_time_sk;
> optional int64 ss_item_sk;
> optional int64 ss_customer_sk;
> optional int64 ss_cdemo_sk;
> optional int64 ss_hdemo_sk;
> optional int64 ss_addr_sk;
> optional int64 ss_store_sk;
> optional int64 ss_promo_sk;
> optional int64 ss_ticket_number;
> optional int64 ss_quantity;
> optional double ss_wholesale_cost;
> optional double ss_list_price;
> optional double ss_sales_price;
> optional double ss_ext_discount_amt;
> optional double ss_ext_sales_price;
> optional double ss_ext_wholesale_cost;
> optional double ss_ext_list_price;
> optional double ss_ext_tax;
> optional double ss_coupon_amt;
> optional double ss_net_paid;
> optional double ss_net_paid_inc_tax;
> optional double ss_net_profit;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4208) Storage plugin configuration persistence not working for Apache Drill
[
https://issues.apache.org/jira/browse/DRILL-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063271#comment-15063271
]
Steven Phillips commented on DRILL-4208:
The properties in drill-override.conf are hierarchical. Since you are already
inside drill.exec, you don't include it in the key. So it should be like this:
drill.exec: {
cluster-id: "drillbits1",
zk.connect: "localhost:2181",
sys.store.provider.local.path = "/home/dev/abc"
}
> Storage plugin configuration persistence not working for Apache Drill
> -
>
> Key: DRILL-4208
> URL: https://issues.apache.org/jira/browse/DRILL-4208
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.3.0
> Environment: Ubuntu 14.0.4
>Reporter: Devender Yadav
> Fix For: Future
>
>
> According to Drill's documentation :
> Drill uses /tmp/drill/sys.storage_plugins to store storage plugin
> configurations. The temporary directory clears when you quit the Drill shell.
> To save your storage plugin configurations from one session to the next, set
> the following option in the drill-override.conf file if you are running Drill
> in embedded mode.
> drill.exec.sys.store.provider.local.path = "/mypath"
> I checked /tmp/drill/sys.storage_plugins, there is some data in this file.
> Then I modified drill-override.conf :
> drill.exec: {
> cluster-id: "drillbits1",
> zk.connect: "localhost:2181",
> drill.exec.sys.store.provider.local.path = "/home/dev/abc"
> }
> I restarted drill & even restarted my machine. Nothing is created at this
> location.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-4201) DrillPushFilterPastProject should allow partial filter pushdown.
[ https://issues.apache.org/jira/browse/DRILL-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Sinha updated DRILL-4201: -- Assignee: Jinfeng Ni (was: Aman Sinha) > DrillPushFilterPastProject should allow partial filter pushdown. > - > > Key: DRILL-4201 > URL: https://issues.apache.org/jira/browse/DRILL-4201 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > Fix For: 1.5.0 > > > Currently, DrillPushFilterPastProjectRule will stop pushing the filter down, > if the filter itself has ITEM or FLATTEN function, or its input reference is > referring to an ITEM or FLATTEN function. However, in case that the filter is > a conjunction of multiple sub-filters, some of them refer to ITEM or FLATTEN > but the other not, then we should allow partial filter to be pushed down. For > instance, > WHERE partition_col > 10 and flatten_output_col = 'ABC'. > The "flatten_output_col" comes from the output of FLATTEN operator, and > therefore flatten_output_col = 'ABC' should not pushed past the project. But > partiion_col > 10 should be pushed down, such that we could trigger the > pruning rule to apply partition pruning. > It would be improve Drill query performance, when the partially pushed filter > leads to partition pruning, or the partially pushed filter results in early > filtering in upstream operator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4201) DrillPushFilterPastProject should allow partial filter pushdown.
[ https://issues.apache.org/jira/browse/DRILL-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063266#comment-15063266 ] ASF GitHub Bot commented on DRILL-4201: --- Github user amansinha100 commented on the pull request: https://github.com/apache/drill/pull/305#issuecomment-165635050 +1 LGTM. > DrillPushFilterPastProject should allow partial filter pushdown. > - > > Key: DRILL-4201 > URL: https://issues.apache.org/jira/browse/DRILL-4201 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Reporter: Jinfeng Ni >Assignee: Aman Sinha > Fix For: 1.5.0 > > > Currently, DrillPushFilterPastProjectRule will stop pushing the filter down, > if the filter itself has ITEM or FLATTEN function, or its input reference is > referring to an ITEM or FLATTEN function. However, in case that the filter is > a conjunction of multiple sub-filters, some of them refer to ITEM or FLATTEN > but the other not, then we should allow partial filter to be pushed down. For > instance, > WHERE partition_col > 10 and flatten_output_col = 'ABC'. > The "flatten_output_col" comes from the output of FLATTEN operator, and > therefore flatten_output_col = 'ABC' should not pushed past the project. But > partiion_col > 10 should be pushed down, such that we could trigger the > pruning rule to apply partition pruning. > It would be improve Drill query performance, when the partially pushed filter > leads to partition pruning, or the partially pushed filter results in early > filtering in upstream operator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4201) DrillPushFilterPastProject should allow partial filter pushdown.
[ https://issues.apache.org/jira/browse/DRILL-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinfeng Ni updated DRILL-4201: -- Assignee: Aman Sinha (was: Jinfeng Ni) > DrillPushFilterPastProject should allow partial filter pushdown. > - > > Key: DRILL-4201 > URL: https://issues.apache.org/jira/browse/DRILL-4201 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Reporter: Jinfeng Ni >Assignee: Aman Sinha > Fix For: 1.5.0 > > > Currently, DrillPushFilterPastProjectRule will stop pushing the filter down, > if the filter itself has ITEM or FLATTEN function, or its input reference is > referring to an ITEM or FLATTEN function. However, in case that the filter is > a conjunction of multiple sub-filters, some of them refer to ITEM or FLATTEN > but the other not, then we should allow partial filter to be pushed down. For > instance, > WHERE partition_col > 10 and flatten_output_col = 'ABC'. > The "flatten_output_col" comes from the output of FLATTEN operator, and > therefore flatten_output_col = 'ABC' should not pushed past the project. But > partiion_col > 10 should be pushed down, such that we could trigger the > pruning rule to apply partition pruning. > It would be improve Drill query performance, when the partially pushed filter > leads to partition pruning, or the partially pushed filter results in early > filtering in upstream operator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DRILL-4210) typo in documentation
[ https://issues.apache.org/jira/browse/DRILL-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kristine Hahn resolved DRILL-4210. -- Resolution: Fixed Fix Version/s: 1.5.0 Fixed. Thanks! > typo in documentation > - > > Key: DRILL-4210 > URL: https://issues.apache.org/jira/browse/DRILL-4210 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Reporter: Devender Yadav >Assignee: Kristine Hahn >Priority: Trivial > Fix For: 1.5.0 > > > Link : > https://drill.apache.org/docs/storage-plugin-registration/#storage-plugin-configuration-persistance > Persistence is spelled wrongly as "Persistance". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (DRILL-4168) NULLIF function documentation should be corrected
[
https://issues.apache.org/jira/browse/DRILL-4168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristine Hahn resolved DRILL-4168.
--
Resolution: Fixed
Fixed. Thanks!
> NULLIF function documentation should be corrected
> -
>
> Key: DRILL-4168
> URL: https://issues.apache.org/jira/browse/DRILL-4168
> Project: Apache Drill
> Issue Type: Bug
> Components: Documentation
>Reporter: Victoria Markman
>Assignee: Kristine Hahn
> Fix For: 1.5.0
>
>
> As of 1.4.0 NULLIF accepts two arguments.
> {code}
> 0: jdbc:drill:schema=dfs> select nullif(c_integer, c_bigint, c_double,
> 123.123) from v1;
> Error: VALIDATION ERROR: From line 1, column 8 to line 1, column 53: Invalid
> number of arguments to function 'NULLIF'. Was expecting 2 arguments
> [Error Id: 93203c41-cc8d-45fe-9e8f-b46233d10ff9 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> Documentation states this:
> {code}
> NULLIF Syntax
> NULLIF ( expr1[, expr2, ... expr_n] ) <-- can only be 2
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (DRILL-4209) Error in hive-to-drill-data-type-mapping documentation
[ https://issues.apache.org/jira/browse/DRILL-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kristine Hahn resolved DRILL-4209. -- Resolution: Fixed Fix Version/s: 1.5.0 Clarified http://drill.apache.org/docs/hive-to-drill-data-type-mapping/ Thanks! > Error in hive-to-drill-data-type-mapping documentation > --- > > Key: DRILL-4209 > URL: https://issues.apache.org/jira/browse/DRILL-4209 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Affects Versions: 1.3.0 >Reporter: Devender Yadav >Assignee: Kristine Hahn > Fix For: 1.5.0 > > > Link: https://drill.apache.org/docs/hive-to-drill-data-type-mapping/ > It says CHAR is supported and LIST, MAP, STRUCT, TIMESTAMP (Unix Epoch > format), UNION are not supported. > While querying a table with char datatype field, Drill client says: > Unsupported Hive data type CHAR. > Following Hive data types are supported in Drill INFORMATION_SCHEMA: > BOOLEAN, BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, DATE, TIMESTAMP, > BINARY, DECIMAL, STRING, VARCHAR, LIST, MAP, STRUCT and UNION > This is really confusing. Please look into this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4205) Simple query hit IndexOutOfBoundException
[
https://issues.apache.org/jira/browse/DRILL-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063177#comment-15063177
]
ASF GitHub Bot commented on DRILL-4205:
---
Github user adeneche commented on the pull request:
https://github.com/apache/drill/pull/306#issuecomment-165624553
+1, LGTM
We should try and add a test to the public test framework, once this PR is
merged
> Simple query hit IndexOutOfBoundException
> --
>
> Key: DRILL-4205
> URL: https://issues.apache.org/jira/browse/DRILL-4205
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Dechang Gu
>Assignee: Parth Chandra
>
> The following query failed due to IOB:
> 0: jdbc:drill:schema=wf_pigprq100> select * from
> `store_sales/part-m-00073.parquet`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: srcIndex: 1048587
> Fragment 0:0
> [Error Id: ad8d2bc0-259f-483c-9024-93865963541e on ucs-node4.perf.lab:31010]
> (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet
> record reader.
> Message:
> Hadoop path: /tpcdsPigParq/SF100/store_sales/part-m-00073.parquet
> Total records read: 135280
> Mock records read: 0
> Records to read: 1424
> Row group index: 0
> Records in row group: 3775712
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message pig_schema {
> optional int64 ss_sold_date_sk;
> optional int64 ss_sold_time_sk;
> optional int64 ss_item_sk;
> optional int64 ss_customer_sk;
> optional int64 ss_cdemo_sk;
> optional int64 ss_hdemo_sk;
> optional int64 ss_addr_sk;
> optional int64 ss_store_sk;
> optional int64 ss_promo_sk;
> optional int64 ss_ticket_number;
> optional int64 ss_quantity;
> optional double ss_wholesale_cost;
> optional double ss_list_price;
> optional double ss_sales_price;
> optional double ss_ext_discount_amt;
> optional double ss_ext_sales_price;
> optional double ss_ext_wholesale_cost;
> optional double ss_ext_list_price;
> optional double ss_ext_tax;
> optional double ss_coupon_amt;
> optional double ss_net_paid;
> optional double ss_net_paid_inc_tax;
> optional double ss_net_profit;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4205) Simple query hit IndexOutOfBoundException
[
https://issues.apache.org/jira/browse/DRILL-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063173#comment-15063173
]
ASF GitHub Bot commented on DRILL-4205:
---
Github user parthchandra commented on the pull request:
https://github.com/apache/drill/pull/306#issuecomment-165624400
The test file for this is 600MB which is too large for a unit test. I would
recommend we add this to the regression test suite.
> Simple query hit IndexOutOfBoundException
> --
>
> Key: DRILL-4205
> URL: https://issues.apache.org/jira/browse/DRILL-4205
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Dechang Gu
>Assignee: Parth Chandra
>
> The following query failed due to IOB:
> 0: jdbc:drill:schema=wf_pigprq100> select * from
> `store_sales/part-m-00073.parquet`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: srcIndex: 1048587
> Fragment 0:0
> [Error Id: ad8d2bc0-259f-483c-9024-93865963541e on ucs-node4.perf.lab:31010]
> (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet
> record reader.
> Message:
> Hadoop path: /tpcdsPigParq/SF100/store_sales/part-m-00073.parquet
> Total records read: 135280
> Mock records read: 0
> Records to read: 1424
> Row group index: 0
> Records in row group: 3775712
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message pig_schema {
> optional int64 ss_sold_date_sk;
> optional int64 ss_sold_time_sk;
> optional int64 ss_item_sk;
> optional int64 ss_customer_sk;
> optional int64 ss_cdemo_sk;
> optional int64 ss_hdemo_sk;
> optional int64 ss_addr_sk;
> optional int64 ss_store_sk;
> optional int64 ss_promo_sk;
> optional int64 ss_ticket_number;
> optional int64 ss_quantity;
> optional double ss_wholesale_cost;
> optional double ss_list_price;
> optional double ss_sales_price;
> optional double ss_ext_discount_amt;
> optional double ss_ext_sales_price;
> optional double ss_ext_wholesale_cost;
> optional double ss_ext_list_price;
> optional double ss_ext_tax;
> optional double ss_coupon_amt;
> optional double ss_net_paid;
> optional double ss_net_paid_inc_tax;
> optional double ss_net_profit;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4182) Support Schema changes for TopN Operator
[ https://issues.apache.org/jira/browse/DRILL-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063027#comment-15063027 ] ASF GitHub Bot commented on DRILL-4182: --- Github user amithadke commented on the pull request: https://github.com/apache/drill/pull/297#issuecomment-165609542 @StevenMPhillips Can you review this change? > Support Schema changes for TopN Operator > > > Key: DRILL-4182 > URL: https://issues.apache.org/jira/browse/DRILL-4182 > Project: Apache Drill > Issue Type: Improvement >Reporter: amit hadke >Assignee: amit hadke > > Support schema changes for top in (order by limit) operator. > Cases: > 1) Changes when schema type changes for order by field. > 2) Schema changes for other fields > 3) When field itself is a union type. > 4) Missing field. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4205) Simple query hit IndexOutOfBoundException
[
https://issues.apache.org/jira/browse/DRILL-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062964#comment-15062964
]
ASF GitHub Bot commented on DRILL-4205:
---
Github user adeneche commented on the pull request:
https://github.com/apache/drill/pull/306#issuecomment-165599123
can we add a unit test ? thanks
> Simple query hit IndexOutOfBoundException
> --
>
> Key: DRILL-4205
> URL: https://issues.apache.org/jira/browse/DRILL-4205
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Dechang Gu
>Assignee: Parth Chandra
>
> The following query failed due to IOB:
> 0: jdbc:drill:schema=wf_pigprq100> select * from
> `store_sales/part-m-00073.parquet`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: srcIndex: 1048587
> Fragment 0:0
> [Error Id: ad8d2bc0-259f-483c-9024-93865963541e on ucs-node4.perf.lab:31010]
> (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet
> record reader.
> Message:
> Hadoop path: /tpcdsPigParq/SF100/store_sales/part-m-00073.parquet
> Total records read: 135280
> Mock records read: 0
> Records to read: 1424
> Row group index: 0
> Records in row group: 3775712
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message pig_schema {
> optional int64 ss_sold_date_sk;
> optional int64 ss_sold_time_sk;
> optional int64 ss_item_sk;
> optional int64 ss_customer_sk;
> optional int64 ss_cdemo_sk;
> optional int64 ss_hdemo_sk;
> optional int64 ss_addr_sk;
> optional int64 ss_store_sk;
> optional int64 ss_promo_sk;
> optional int64 ss_ticket_number;
> optional int64 ss_quantity;
> optional double ss_wholesale_cost;
> optional double ss_list_price;
> optional double ss_sales_price;
> optional double ss_ext_discount_amt;
> optional double ss_ext_sales_price;
> optional double ss_ext_wholesale_cost;
> optional double ss_ext_list_price;
> optional double ss_ext_tax;
> optional double ss_coupon_amt;
> optional double ss_net_paid;
> optional double ss_net_paid_inc_tax;
> optional double ss_net_profit;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4187) Introduce a state to separate queries pending execution from those pending in the queue.
[ https://issues.apache.org/jira/browse/DRILL-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062709#comment-15062709 ] Hanifi Gunes commented on DRILL-4187: - This patch should have no effect on reading old profiles at a terminal state. Since enum order in proto is maintained I would think this patch would not cause any problem for old PENDING new STARTING non-terminal state as well. > Introduce a state to separate queries pending execution from those pending in > the queue. > > > Key: DRILL-4187 > URL: https://issues.apache.org/jira/browse/DRILL-4187 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Hanifi Gunes >Assignee: Mehant Baid > Fix For: 1.5.0 > > > Currently queries pending in the queue are not listed in the web UI besides > we use the state PENDING to mean pending executions. This issue proposes i) > to list enqueued queries in the web UI ii) to introduce a new state for > queries sitting at the queue, differentiating then from those pending > execution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4205) Simple query hit IndexOutOfBoundException
[
https://issues.apache.org/jira/browse/DRILL-4205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062693#comment-15062693
]
ASF GitHub Bot commented on DRILL-4205:
---
GitHub user parthchandra opened a pull request:
https://github.com/apache/drill/pull/306
DRILL-4205: Reset readStart every time a new page is read.
The readStart index was not being reset every time a new page was read and
caused an array index out of bounds.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/parthchandra/incubator-drill DRILL-4205
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/306.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #306
commit aaa6bb850b9de1e86afe2e0a5afad90d753fca04
Author: Parth Chandra
Date: 2015-12-16T23:25:26Z
DRILL-4205: Reset readStart every time a new page is read.
> Simple query hit IndexOutOfBoundException
> --
>
> Key: DRILL-4205
> URL: https://issues.apache.org/jira/browse/DRILL-4205
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.4.0
>Reporter: Dechang Gu
>Assignee: Parth Chandra
>
> The following query failed due to IOB:
> 0: jdbc:drill:schema=wf_pigprq100> select * from
> `store_sales/part-m-00073.parquet`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: srcIndex: 1048587
> Fragment 0:0
> [Error Id: ad8d2bc0-259f-483c-9024-93865963541e on ucs-node4.perf.lab:31010]
> (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet
> record reader.
> Message:
> Hadoop path: /tpcdsPigParq/SF100/store_sales/part-m-00073.parquet
> Total records read: 135280
> Mock records read: 0
> Records to read: 1424
> Row group index: 0
> Records in row group: 3775712
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message pig_schema {
> optional int64 ss_sold_date_sk;
> optional int64 ss_sold_time_sk;
> optional int64 ss_item_sk;
> optional int64 ss_customer_sk;
> optional int64 ss_cdemo_sk;
> optional int64 ss_hdemo_sk;
> optional int64 ss_addr_sk;
> optional int64 ss_store_sk;
> optional int64 ss_promo_sk;
> optional int64 ss_ticket_number;
> optional int64 ss_quantity;
> optional double ss_wholesale_cost;
> optional double ss_list_price;
> optional double ss_sales_price;
> optional double ss_ext_discount_amt;
> optional double ss_ext_sales_price;
> optional double ss_ext_wholesale_cost;
> optional double ss_ext_list_price;
> optional double ss_ext_tax;
> optional double ss_coupon_amt;
> optional double ss_net_paid;
> optional double ss_net_paid_inc_tax;
> optional double ss_net_profit;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DRILL-4201) DrillPushFilterPastProject should allow partial filter pushdown.
[ https://issues.apache.org/jira/browse/DRILL-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062688#comment-15062688 ] ASF GitHub Bot commented on DRILL-4201: --- GitHub user jinfengni opened a pull request: https://github.com/apache/drill/pull/305 DRILL-4201 : Allow partial filter to be pushed down project for bette… …r performance. Partial filter pushdown has performance benefits because: 1) enable partition pruning, if the pushed down involves partitioning columns, 2) allow the filter to be applied in upper stream. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinfengni/incubator-drill DRILL-4201 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/305.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #305 commit 68203ad035f65d1708ce228df432d5b23f4af3ba Author: Jinfeng Ni Date: 2015-12-12T00:00:13Z DRILL-4201 : Allow partial filter to be pushed down project for better performance. Partial filter pushdown has performance benefits because: 1) enable partition pruning, if the pushed down involves partitioning columns, 2) allow the filter to be applied in upper stream. > DrillPushFilterPastProject should allow partial filter pushdown. > - > > Key: DRILL-4201 > URL: https://issues.apache.org/jira/browse/DRILL-4201 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > Fix For: 1.5.0 > > > Currently, DrillPushFilterPastProjectRule will stop pushing the filter down, > if the filter itself has ITEM or FLATTEN function, or its input reference is > referring to an ITEM or FLATTEN function. However, in case that the filter is > a conjunction of multiple sub-filters, some of them refer to ITEM or FLATTEN > but the other not, then we should allow partial filter to be pushed down. For > instance, > WHERE partition_col > 10 and flatten_output_col = 'ABC'. > The "flatten_output_col" comes from the output of FLATTEN operator, and > therefore flatten_output_col = 'ABC' should not pushed past the project. But > partiion_col > 10 should be pushed down, such that we could trigger the > pruning rule to apply partition pruning. > It would be improve Drill query performance, when the partially pushed filter > leads to partition pruning, or the partially pushed filter results in early > filtering in upstream operator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4201) DrillPushFilterPastProject should allow partial filter pushdown.
[ https://issues.apache.org/jira/browse/DRILL-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062690#comment-15062690 ] ASF GitHub Bot commented on DRILL-4201: --- Github user jinfengni commented on the pull request: https://github.com/apache/drill/pull/305#issuecomment-165567570 @amansinha100 , could you please review the patch for DRILL-4201? Thanks! > DrillPushFilterPastProject should allow partial filter pushdown. > - > > Key: DRILL-4201 > URL: https://issues.apache.org/jira/browse/DRILL-4201 > Project: Apache Drill > Issue Type: Bug > Components: Query Planning & Optimization >Reporter: Jinfeng Ni >Assignee: Jinfeng Ni > Fix For: 1.5.0 > > > Currently, DrillPushFilterPastProjectRule will stop pushing the filter down, > if the filter itself has ITEM or FLATTEN function, or its input reference is > referring to an ITEM or FLATTEN function. However, in case that the filter is > a conjunction of multiple sub-filters, some of them refer to ITEM or FLATTEN > but the other not, then we should allow partial filter to be pushed down. For > instance, > WHERE partition_col > 10 and flatten_output_col = 'ABC'. > The "flatten_output_col" comes from the output of FLATTEN operator, and > therefore flatten_output_col = 'ABC' should not pushed past the project. But > partiion_col > 10 should be pushed down, such that we could trigger the > pruning rule to apply partition pruning. > It would be improve Drill query performance, when the partially pushed filter > leads to partition pruning, or the partially pushed filter results in early > filtering in upstream operator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4134) Incorporate remaining patches from DRILL-1942 Allocator refactor
[
https://issues.apache.org/jira/browse/DRILL-4134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062685#comment-15062685
]
ASF GitHub Bot commented on DRILL-4134:
---
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/283#discussion_r47953695
--- Diff:
exec/memory/base/src/main/java/org/apache/drill/exec/memory/Accountant.java ---
@@ -0,0 +1,269 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.memory;
+
+import java.util.concurrent.atomic.AtomicLong;
+
+import javax.annotation.concurrent.ThreadSafe;
+
+import org.apache.drill.exec.exception.OutOfMemoryException;
+
+import com.google.common.base.Preconditions;
+
+/**
+ * Provides a concurrent way to manage account for memory usage without
locking. Used as basis for Allocators. All
+ * operations are threadsafe (except for close).
+ */
+@ThreadSafe
+class Accountant implements AutoCloseable {
+ // private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(Accountant.class);
+
+ /**
+ * The parent allocator
+ */
+ protected final Accountant parent;
+
+ /**
+ * The amount of memory reserved for this allocator. Releases below this
amount of memory will not be returned to the
+ * parent Accountant until this Accountant is closed.
+ */
+ protected final long reservation;
+
+ private final AtomicLong peakAllocation = new AtomicLong();
+
+ /**
+ * Maximum local memory that can be held. This can be externally
updated. Changing it won't cause past memory to
+ * change but will change responses to future allocation efforts
+ */
+ private final AtomicLong allocationLimit = new AtomicLong();
+
+ /**
+ * Currently allocated amount of memory;
+ */
+ private final AtomicLong locallyHeldMemory = new AtomicLong();
+
+ public Accountant(Accountant parent, long reservation, long
maxAllocation) {
+Preconditions.checkArgument(reservation >= 0, "The initial reservation
size must be non-negative.");
+Preconditions.checkArgument(maxAllocation >= 0, "The maximum
allocation limit must be non-negative.");
+Preconditions.checkArgument(reservation <= maxAllocation,
+"The initial reservation size must be <= the maximum allocation.");
+Preconditions.checkArgument(reservation == 0 || parent != null, "The
root accountant can't reserve memory.");
+
+this.parent = parent;
+this.reservation = reservation;
+this.allocationLimit.set(maxAllocation);
+
+if (reservation != 0) {
+ // we will allocate a reservation from our parent.
+ final AllocationOutcome outcome = parent.allocateBytes(reservation);
+ if (!outcome.isOk()) {
+throw new OutOfMemoryException(String.format(
+"Failure trying to allocate initial reservation for Allocator.
"
++ "Attempted to allocate %d bytes and received an outcome
of %s.", reservation, outcome.name()));
+ }
+}
+ }
+
+ /**
+ * Attempt to allocate the requested amount of memory. Either completely
succeeds or completely fails. Constructs a a
+ * log of delta
+ *
+ * If it fails, no changes are made to accounting.
+ *
+ * @param size
+ * The amount of memory to reserve in bytes.
+ * @return True if the allocation was successful, false if the
allocation failed.
+ */
+ AllocationOutcome allocateBytes(long size) {
+final AllocationOutcome outcome = allocate(size, true, false);
+if (!outcome.isOk()) {
+ releaseBytes(size);
+}
+return outcome;
+ }
+
+ private void updatePeak() {
+final long currentMemory = locallyHeldMemory.get();
+
+long previousPeak;
+do {
+ previousPeak = peakAllocation.get();
[jira] [Commented] (DRILL-4187) Introduce a state to separate queries pending execution from those pending in the queue.
[ https://issues.apache.org/jira/browse/DRILL-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062642#comment-15062642 ] Jacques Nadeau commented on DRILL-4187: --- One question I have for this change is what happens to profiles that have already been written. Are they still readable? I think they would be since a written profile would(should?) be in a terminal state (and we're only renaming a non-terminal state). However, I think it is probably important to figure that out and make sure people can continue to read their old profiles. > Introduce a state to separate queries pending execution from those pending in > the queue. > > > Key: DRILL-4187 > URL: https://issues.apache.org/jira/browse/DRILL-4187 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Hanifi Gunes >Assignee: Mehant Baid > Fix For: 1.5.0 > > > Currently queries pending in the queue are not listed in the web UI besides > we use the state PENDING to mean pending executions. This issue proposes i) > to list enqueued queries in the web UI ii) to introduce a new state for > queries sitting at the queue, differentiating then from those pending > execution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4187) Introduce a state to separate queries pending execution from those pending in the queue.
[ https://issues.apache.org/jira/browse/DRILL-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062622#comment-15062622 ] Hanifi Gunes commented on DRILL-4187: - [~mehant] please review. > Introduce a state to separate queries pending execution from those pending in > the queue. > > > Key: DRILL-4187 > URL: https://issues.apache.org/jira/browse/DRILL-4187 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Hanifi Gunes >Assignee: Mehant Baid > Fix For: 1.5.0 > > > Currently queries pending in the queue are not listed in the web UI besides > we use the state PENDING to mean pending executions. This issue proposes i) > to list enqueued queries in the web UI ii) to introduce a new state for > queries sitting at the queue, differentiating then from those pending > execution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4187) Introduce a state to separate queries pending execution from those pending in the queue.
[ https://issues.apache.org/jira/browse/DRILL-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanifi Gunes updated DRILL-4187: Assignee: Mehant Baid (was: Hanifi Gunes) > Introduce a state to separate queries pending execution from those pending in > the queue. > > > Key: DRILL-4187 > URL: https://issues.apache.org/jira/browse/DRILL-4187 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Hanifi Gunes >Assignee: Mehant Baid > Fix For: 1.5.0 > > > Currently queries pending in the queue are not listed in the web UI besides > we use the state PENDING to mean pending executions. This issue proposes i) > to list enqueued queries in the web UI ii) to introduce a new state for > queries sitting at the queue, differentiating then from those pending > execution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-2848) Disable decimal data type by default
[ https://issues.apache.org/jira/browse/DRILL-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062508#comment-15062508 ] Andries Engelbrecht commented on DRILL-2848: It is feasible to enable decimal by default in future versions? A number of BI and Analytical Software tools that work with Drill requested this. > Disable decimal data type by default > > > Key: DRILL-2848 > URL: https://issues.apache.org/jira/browse/DRILL-2848 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Data Types >Reporter: Mehant Baid >Assignee: Jinfeng Ni >Priority: Critical > Fix For: 1.0.0 > > Attachments: DRILL-2848-part1.patch, DRILL-2848-part2.patch > > > Due to the difference in the storage format of decimal data type in parquet > versus the in-memory format within Drill using the decimal data type is not > performant. Also some of the rules for calculating the scale and precision > need to be changed. These two concerns will be addressed post 1.0.0 release > and to prevent users from running into this we are disabling decimal data > type by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-3429) DrillAvgVarianceConvertlet may produce wrong results while rewriting stddev, variance
[
https://issues.apache.org/jira/browse/DRILL-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062498#comment-15062498
]
Victoria Markman commented on DRILL-3429:
-
Verified fixed in 1.4.0
{code}
#Tue Dec 15 23:00:30 UTC 2015
git.commit.id.abbrev=2fbe668
git.commit.user.email=par...@apache.org
git.commit.message.full=DRILL-4152\: Add trace logging to Parquet reader for
performance tuning.\n
git.commit.id=2fbe668e8dcb3d06f345feb32a3d2ab1559f3645
{code}
Tests added in: Functional/window_functions/statistical_aggregates
> DrillAvgVarianceConvertlet may produce wrong results while rewriting stddev,
> variance
> -
>
> Key: DRILL-3429
> URL: https://issues.apache.org/jira/browse/DRILL-3429
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Mehant Baid
>Assignee: Mehant Baid
>Priority: Critical
> Fix For: 1.3.0
>
> Attachments: DRILL-3429.patch
>
>
> DrillAvgVarianceConvertlet currently rewrites aggregate functions like avg,
> stddev, variance to simple computations.
> Eg:
> Stddev( x ) => power(
> (sum(x * x) - sum( x ) * sum( x ) / count( x ))
> / count( x ),
> .5)
> Consider the case when the input is an integer. Now the rewrite contains
> multiplication and division, which will bind to functions that operate on
> integers however the expected result should be a double and since double has
> more precision than integer we should be operating on double during the
> multiplication and division.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (DRILL-3429) DrillAvgVarianceConvertlet may produce wrong results while rewriting stddev, variance
[ https://issues.apache.org/jira/browse/DRILL-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victoria Markman closed DRILL-3429. --- > DrillAvgVarianceConvertlet may produce wrong results while rewriting stddev, > variance > - > > Key: DRILL-3429 > URL: https://issues.apache.org/jira/browse/DRILL-3429 > Project: Apache Drill > Issue Type: Bug >Reporter: Mehant Baid >Assignee: Mehant Baid >Priority: Critical > Fix For: 1.3.0 > > Attachments: DRILL-3429.patch > > > DrillAvgVarianceConvertlet currently rewrites aggregate functions like avg, > stddev, variance to simple computations. > Eg: > Stddev( x ) => power( > (sum(x * x) - sum( x ) * sum( x ) / count( x )) > / count( x ), > .5) > Consider the case when the input is an integer. Now the rewrite contains > multiplication and division, which will bind to functions that operate on > integers however the expected result should be a double and since double has > more precision than integer we should be operating on double during the > multiplication and division. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (DRILL-4000) In all non-root fragments, Drill recreates storage plugin instances for every minor fragment
[ https://issues.apache.org/jira/browse/DRILL-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dechang Gu closed DRILL-4000. - Verified, and no issue is seen. LGTM. > In all non-root fragments, Drill recreates storage plugin instances for every > minor fragment > > > Key: DRILL-4000 > URL: https://issues.apache.org/jira/browse/DRILL-4000 > Project: Apache Drill > Issue Type: Bug >Reporter: Jacques Nadeau >Assignee: Jacques Nadeau > Fix For: 1.3.0 > > > Drill is creating ephemeral storage plugin instances when a plan is > deserialized. As such, every minor fragment of a query has Drill create a > separate storage plugin instance. Depending on the cost of storage plugin > creation, this could be quite expensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (DRILL-4210) typo in documentation
[ https://issues.apache.org/jira/browse/DRILL-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kristine Hahn reassigned DRILL-4210: Assignee: Kristine Hahn > typo in documentation > - > > Key: DRILL-4210 > URL: https://issues.apache.org/jira/browse/DRILL-4210 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Reporter: Devender Yadav >Assignee: Kristine Hahn >Priority: Trivial > > Link : > https://drill.apache.org/docs/storage-plugin-registration/#storage-plugin-configuration-persistance > Persistence is spelled wrongly as "Persistance". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (DRILL-4168) NULLIF function documentation should be corrected
[
https://issues.apache.org/jira/browse/DRILL-4168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristine Hahn reassigned DRILL-4168:
Assignee: Kristine Hahn
> NULLIF function documentation should be corrected
> -
>
> Key: DRILL-4168
> URL: https://issues.apache.org/jira/browse/DRILL-4168
> Project: Apache Drill
> Issue Type: Bug
> Components: Documentation
>Reporter: Victoria Markman
>Assignee: Kristine Hahn
> Fix For: 1.5.0
>
>
> As of 1.4.0 NULLIF accepts two arguments.
> {code}
> 0: jdbc:drill:schema=dfs> select nullif(c_integer, c_bigint, c_double,
> 123.123) from v1;
> Error: VALIDATION ERROR: From line 1, column 8 to line 1, column 53: Invalid
> number of arguments to function 'NULLIF'. Was expecting 2 arguments
> [Error Id: 93203c41-cc8d-45fe-9e8f-b46233d10ff9 on atsqa4-133.qa.lab:31010]
> (state=,code=0)
> {code}
> Documentation states this:
> {code}
> NULLIF Syntax
> NULLIF ( expr1[, expr2, ... expr_n] ) <-- can only be 2
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (DRILL-4209) Error in hive-to-drill-data-type-mapping documentation
[ https://issues.apache.org/jira/browse/DRILL-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kristine Hahn reassigned DRILL-4209: Assignee: Kristine Hahn > Error in hive-to-drill-data-type-mapping documentation > --- > > Key: DRILL-4209 > URL: https://issues.apache.org/jira/browse/DRILL-4209 > Project: Apache Drill > Issue Type: Bug > Components: Documentation >Affects Versions: 1.3.0 >Reporter: Devender Yadav >Assignee: Kristine Hahn > > Link: https://drill.apache.org/docs/hive-to-drill-data-type-mapping/ > It says CHAR is supported and LIST, MAP, STRUCT, TIMESTAMP (Unix Epoch > format), UNION are not supported. > While querying a table with char datatype field, Drill client says: > Unsupported Hive data type CHAR. > Following Hive data types are supported in Drill INFORMATION_SCHEMA: > BOOLEAN, BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, DATE, TIMESTAMP, > BINARY, DECIMAL, STRING, VARCHAR, LIST, MAP, STRUCT and UNION > This is really confusing. Please look into this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (DRILL-4206) Move all_text_mode and read_numbers_as_double options to the JSON format plugin and out of system/session
[ https://issues.apache.org/jira/browse/DRILL-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062123#comment-15062123 ] John Omernik commented on DRILL-4206: - This also relates to my JIRA https://issues.apache.org/jira/browse/DRILL-4130, I just made more of a high level association in that I didn't want to only specify those two settings. (I.e. are there other settings that should be moved closer to the plugin because of the ability to use select with options?) > Move all_text_mode and read_numbers_as_double options to the JSON format > plugin and out of system/session > - > > Key: DRILL-4206 > URL: https://issues.apache.org/jira/browse/DRILL-4206 > Project: Apache Drill > Issue Type: Improvement >Reporter: Jason Altekruse >Assignee: Jason Altekruse > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (DRILL-4211) Inconsistent results from a joined sql statement to postgres tables
[
https://issues.apache.org/jira/browse/DRILL-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Hamilton-Smith updated DRILL-4211:
-
Target Version/s: Future
> Inconsistent results from a joined sql statement to postgres tables
> ---
>
> Key: DRILL-4211
> URL: https://issues.apache.org/jira/browse/DRILL-4211
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
>Affects Versions: 1.3.0
> Environment: Postgres db stroage
>Reporter: Robert Hamilton-Smith
> Labels: newbie
>
> When making an sql statement that incorporates a join to a table and then a
> self join to that table to get a parent value , Drill brings back
> inconsistent results.
> Here is the sql in postgres with correct output:
> {code:sql}
> select trx.categoryguid,
> cat.categoryname, w1.categoryname as parentcat
> from transactions trx
> join categories cat on (cat.CATEGORYGUID = trx.CATEGORYGUID)
> join categories w1 on (cat.categoryparentguid = w1.categoryguid)
> where cat.categoryparentguid IS NOT NULL;
> {code}
> Output:
> ||categoryid||categoryname||parentcategory||
> |id1|restaurants|food&Dining|
> |id1|restaurants|food&Dining|
> |id2|Coffee Shops|food&Dining|
> |id2|Coffee Shops|food&Dining|
> When run in Drill with correct storage prefix:
> {code:sql}
> select trx.categoryguid,
> cat.categoryname, w1.categoryname as parentcat
> from db.schema.transactions trx
> join db.schema.categories cat on (cat.CATEGORYGUID = trx.CATEGORYGUID)
> join db.schema.wpfm_categories w1 on (cat.categoryparentguid =
> w1.categoryguid)
> where cat.categoryparentguid IS NOT NULL
> {code}
> Results are:
> ||categoryid||categoryname||parentcategory||
> |id1|restaurants|null|
> |id1|restaurants|null|
> |id2|Coffee Shops|null|
> |id2|Coffee Shops|null|
> Physical plan is:
> {code:sql}
> 00-00Screen : rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50)
> categoryname, VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost =
> {110.0 rows, 110.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64293
> 00-01 Project(categoryguid=[$0], categoryname=[$1], parentcat=[$2]) :
> rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) categoryname,
> VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = {100.0 rows,
> 100.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64292
> 00-02Project(categoryguid=[$9], categoryname=[$41], parentcat=[$47])
> : rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) categoryname,
> VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = {100.0 rows,
> 100.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64291
> 00-03 Jdbc(sql=[SELECT *
> FROM "public"."transactions"
> INNER JOIN (SELECT *
> FROM "public"."categories"
> WHERE "categoryparentguid" IS NOT NULL) AS "t" ON
> "transactions"."categoryguid" = "t"."categoryguid"
> INNER JOIN "public"."categories" AS "categories0" ON "t"."categoryparentguid"
> = "categories0"."categoryguid"]) : rowType = RecordType(VARCHAR(255)
> transactionguid, VARCHAR(255) relatedtransactionguid, VARCHAR(255)
> transactioncode, DECIMAL(1, 0) transactionpending, VARCHAR(50)
> transactionrefobjecttype, VARCHAR(255) transactionrefobjectguid,
> VARCHAR(1024) transactionrefobjectvalue, TIMESTAMP(6) transactiondate,
> VARCHAR(256) transactiondescription, VARCHAR(50) categoryguid, VARCHAR(3)
> transactioncurrency, DECIMAL(15, 3) transactionoldbalance, DECIMAL(13, 3)
> transactionamount, DECIMAL(15, 3) transactionnewbalance, VARCHAR(512)
> transactionnotes, DECIMAL(2, 0) transactioninstrumenttype, VARCHAR(20)
> transactioninstrumentsubtype, VARCHAR(20) transactioninstrumentcode,
> VARCHAR(50) transactionorigpartyguid, VARCHAR(255)
> transactionorigaccountguid, VARCHAR(50) transactionrecpartyguid, VARCHAR(255)
> transactionrecaccountguid, VARCHAR(256) transactionstatementdesc, DECIMAL(1,
> 0) transactionsplit, DECIMAL(1, 0) transactionduplicated, DECIMAL(1, 0)
> transactionrecategorized, TIMESTAMP(6) transactioncreatedat, TIMESTAMP(6)
> transactionupdatedat, VARCHAR(50) transactionmatrulerefobjtype, VARCHAR(50)
> transactionmatrulerefobjguid, VARCHAR(50) transactionmatrulerefobjvalue,
> VARCHAR(50) transactionuserruleguid, DECIMAL(2, 0) transactionsplitorder,
> TIMESTAMP(6) transactionprocessedat, TIMESTAMP(6)
> transactioncategoryassignat, VARCHAR(50) transactionsystemcategoryguid,
> VARCHAR(50) transactionorigmandateid, VARCHAR(100) fingerprint, VARCHAR(50)
> categoryguid0, VARCHAR(50) categoryparentguid, DECIMAL(3, 0) categorytype,
> VARCHAR(50) categoryname, VARCHAR(50) categorydescription, VARCHAR(50)
> partyguid, VARCHAR(50) categoryguid1, VARCHAR(50) categoryparentguid0,
> DECIMAL(3, 0) categorytype0, VARCHAR(50) categoryname0, VARCHAR(50)
> categorydescr
[jira] [Created] (DRILL-4211) Inconsistent results from a joined sql statement to postgres tables
Robert Hamilton-Smith created DRILL-4211:
Summary: Inconsistent results from a joined sql statement to
postgres tables
Key: DRILL-4211
URL: https://issues.apache.org/jira/browse/DRILL-4211
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Affects Versions: 1.3.0
Environment: Postgres db stroage
Reporter: Robert Hamilton-Smith
When making an sql statement that incorporates a join to a table and then a
self join to that table to get a parent value , Drill brings back inconsistent
results.
Here is the sql in postgres with correct output:
{code:sql}
select trx.categoryguid,
cat.categoryname, w1.categoryname as parentcat
from transactions trx
join categories cat on (cat.CATEGORYGUID = trx.CATEGORYGUID)
join categories w1 on (cat.categoryparentguid = w1.categoryguid)
where cat.categoryparentguid IS NOT NULL;
{code}
Output:
||categoryid||categoryname||parentcategory||
|id1|restaurants|food&Dining|
|id1|restaurants|food&Dining|
|id2|Coffee Shops|food&Dining|
|id2|Coffee Shops|food&Dining|
When run in Drill with correct storage prefix:
{code:sql}
select trx.categoryguid,
cat.categoryname, w1.categoryname as parentcat
from db.schema.transactions trx
join db.schema.categories cat on (cat.CATEGORYGUID = trx.CATEGORYGUID)
join db.schema.wpfm_categories w1 on (cat.categoryparentguid = w1.categoryguid)
where cat.categoryparentguid IS NOT NULL
{code}
Results are:
||categoryid||categoryname||parentcategory||
|id1|restaurants|null|
|id1|restaurants|null|
|id2|Coffee Shops|null|
|id2|Coffee Shops|null|
Physical plan is:
{code:sql}
00-00Screen : rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50)
categoryname, VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost =
{110.0 rows, 110.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64293
00-01 Project(categoryguid=[$0], categoryname=[$1], parentcat=[$2]) :
rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) categoryname,
VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = {100.0 rows, 100.0
cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64292
00-02Project(categoryguid=[$9], categoryname=[$41], parentcat=[$47]) :
rowType = RecordType(VARCHAR(50) categoryguid, VARCHAR(50) categoryname,
VARCHAR(50) parentcat): rowcount = 100.0, cumulative cost = {100.0 rows, 100.0
cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64291
00-03 Jdbc(sql=[SELECT *
FROM "public"."transactions"
INNER JOIN (SELECT *
FROM "public"."categories"
WHERE "categoryparentguid" IS NOT NULL) AS "t" ON "transactions"."categoryguid"
= "t"."categoryguid"
INNER JOIN "public"."categories" AS "categories0" ON "t"."categoryparentguid" =
"categories0"."categoryguid"]) : rowType = RecordType(VARCHAR(255)
transactionguid, VARCHAR(255) relatedtransactionguid, VARCHAR(255)
transactioncode, DECIMAL(1, 0) transactionpending, VARCHAR(50)
transactionrefobjecttype, VARCHAR(255) transactionrefobjectguid, VARCHAR(1024)
transactionrefobjectvalue, TIMESTAMP(6) transactiondate, VARCHAR(256)
transactiondescription, VARCHAR(50) categoryguid, VARCHAR(3)
transactioncurrency, DECIMAL(15, 3) transactionoldbalance, DECIMAL(13, 3)
transactionamount, DECIMAL(15, 3) transactionnewbalance, VARCHAR(512)
transactionnotes, DECIMAL(2, 0) transactioninstrumenttype, VARCHAR(20)
transactioninstrumentsubtype, VARCHAR(20) transactioninstrumentcode,
VARCHAR(50) transactionorigpartyguid, VARCHAR(255) transactionorigaccountguid,
VARCHAR(50) transactionrecpartyguid, VARCHAR(255) transactionrecaccountguid,
VARCHAR(256) transactionstatementdesc, DECIMAL(1, 0) transactionsplit,
DECIMAL(1, 0) transactionduplicated, DECIMAL(1, 0) transactionrecategorized,
TIMESTAMP(6) transactioncreatedat, TIMESTAMP(6) transactionupdatedat,
VARCHAR(50) transactionmatrulerefobjtype, VARCHAR(50)
transactionmatrulerefobjguid, VARCHAR(50) transactionmatrulerefobjvalue,
VARCHAR(50) transactionuserruleguid, DECIMAL(2, 0) transactionsplitorder,
TIMESTAMP(6) transactionprocessedat, TIMESTAMP(6) transactioncategoryassignat,
VARCHAR(50) transactionsystemcategoryguid, VARCHAR(50)
transactionorigmandateid, VARCHAR(100) fingerprint, VARCHAR(50) categoryguid0,
VARCHAR(50) categoryparentguid, DECIMAL(3, 0) categorytype, VARCHAR(50)
categoryname, VARCHAR(50) categorydescription, VARCHAR(50) partyguid,
VARCHAR(50) categoryguid1, VARCHAR(50) categoryparentguid0, DECIMAL(3, 0)
categorytype0, VARCHAR(50) categoryname0, VARCHAR(50) categorydescription0,
VARCHAR(50) partyguid0): rowcount = 100.0, cumulative cost = {100.0 rows, 100.0
cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 64259
{code}
I worked around it by creating a view on postgres but not ideal. Thanks in
advance.
First Drill Jira Bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (DRILL-4128) null pointer at org.apache.drill.exec.vector.accessor.AbstractSqlAccessor.getString(AbstractSqlAccessor.java:101)
[
https://issues.apache.org/jira/browse/DRILL-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devender Yadav updated DRILL-4128:
---
Fix Version/s: (was: 1.4.0)
Future
> null pointer at
> org.apache.drill.exec.vector.accessor.AbstractSqlAccessor.getString(AbstractSqlAccessor.java:101)
> -
>
> Key: DRILL-4128
> URL: https://issues.apache.org/jira/browse/DRILL-4128
> Project: Apache Drill
> Issue Type: Bug
> Components: Client - JDBC
>Affects Versions: 1.0.0, 1.1.0, 1.2.0, 1.3.0
>Reporter: Devender Yadav
>Priority: Blocker
> Fix For: Future
>
>
> Below mentioned method is throwing null pointer becaue getObject(rowOffset)
> returns null for null values & null.toString() is throwing null pointer.
> @Override
> public String getString(int rowOffset) throws InvalidAccessException{
> return getObject(rowOffset).toString();
> }
> It should be like:
> @Override
> public String getString(int rowOffset) throws InvalidAccessException{
> return getObject(rowOffset)==null? null:getObject(rowOffset).toString();
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
