[jira] [Commented] (DRILL-7341) Vector reAlloc may fails after exchange.
[ https://issues.apache.org/jira/browse/DRILL-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903531#comment-16903531 ] ASF GitHub Bot commented on DRILL-7341: --- weijietong commented on issue #1838: DRILL-7341: Vector reAlloc may fails after exchange URL: https://github.com/apache/drill/pull/1838#issuecomment-519762685 LGTM +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Vector reAlloc may fails after exchange. > > > Key: DRILL-7341 > URL: https://issues.apache.org/jira/browse/DRILL-7341 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.16.0 >Reporter: Oleg Zinoviev >Priority: Major > Attachments: stacktrace.log > > > There are several methods that modify the BaseDataValueVector#data field. > Some of them, such as BaseDataValueVector#exchange, do not change > allocationSizeInBytes. > Therefore, if BaseDataValueVector#exchange was executed for vectors with > different size, *ValueVector#reAlloc may create a buffer of insufficient size. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7341) Vector reAlloc may fails after exchange.
[
https://issues.apache.org/jira/browse/DRILL-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903528#comment-16903528
]
ASF GitHub Bot commented on DRILL-7341:
---
weijietong commented on pull request #1838: DRILL-7341: Vector reAlloc may
fails after exchange
URL: https://github.com/apache/drill/pull/1838#discussion_r312317242
##
File path: exec/vector/src/main/codegen/templates/FixedValueVectors.java
##
@@ -210,10 +210,16 @@ public void reAlloc() {
// a zero-length buffer. Instead, just allocate a 256 byte
// buffer if we start at 0.
-final long newAllocationSize = allocationSizeInBytes == 0
+long newAllocationSize = allocationSizeInBytes == 0
? 256
: allocationSizeInBytes * 2L;
+// Some operations, such as Value Vector#exchange, can be change DrillBuf
data field without corresponding allocation size changes.
+// Check that the size of the allocation is sufficient to copy the old
buffer.
+while (newAllocationSize < data.capacity()) {
Review comment:
The change is to keep backward compatible. The reason I guess is that most
of the fixed VV is number type. Initializing to zero is to keep the java
default value convention.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Vector reAlloc may fails after exchange.
>
>
> Key: DRILL-7341
> URL: https://issues.apache.org/jira/browse/DRILL-7341
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: stacktrace.log
>
>
> There are several methods that modify the BaseDataValueVector#data field.
> Some of them, such as BaseDataValueVector#exchange, do not change
> allocationSizeInBytes.
> Therefore, if BaseDataValueVector#exchange was executed for vectors with
> different size, *ValueVector#reAlloc may create a buffer of insufficient size.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7222) Visualize estimated and actual row counts for a query
[
https://issues.apache.org/jira/browse/DRILL-7222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903274#comment-16903274
]
ASF GitHub Bot commented on DRILL-7222:
---
kkhatua commented on pull request #1779: DRILL-7222: Visualize estimated and
actual row counts for a query
URL: https://github.com/apache/drill/pull/1779#discussion_r312214145
##
File path: exec/java-exec/src/main/resources/rest/profile/profile.ftl
##
@@ -587,6 +622,49 @@
if (e.target.form)
<#if
model.isOnlyImpersonationEnabled()>doSubmitQueryWithUserName()<#else>doSubmitQueryWithAutoLimit();
});
+
+// Convert scientific to Decimal [Ref:
https://gist.github.com/jiggzson/b5f489af9ad931e3d186]
+function scientificToDecimal(num) {
Review comment:
Yep. Verified it. Thanks for pointing it out.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Visualize estimated and actual row counts for a query
> -
>
> Key: DRILL-7222
> URL: https://issues.apache.org/jira/browse/DRILL-7222
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.16.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting, user-experience
> Fix For: 1.17.0
>
>
> With statistics in place, it would be useful to have the *estimated* rowcount
> along side the *actual* rowcount query profile's operator overview.
> We can extract this from the Physical Plan section of the profile.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Resolved] (DRILL-7321) split function doesn't work without from
[
https://issues.apache.org/jira/browse/DRILL-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi resolved DRILL-7321.
Resolution: Fixed
Fix Version/s: 1.17.0
Fixed in the sope of DRILL-7337.
> split function doesn't work without from
>
>
> Key: DRILL-7321
> URL: https://issues.apache.org/jira/browse/DRILL-7321
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.16.0
>Reporter: benj
>Assignee: Volodymyr Vysotskyi
>Priority: Minor
> Fix For: 1.17.0
>
>
> {code:java}
> SELECT upper('foo') AS a /* OK */;
> +-+
> | a |
> +-+
> | foo |
> +-+
> {code}
> but
> {code:java}
> SELECT split('foo,bar,buz',',') AS a /* NOK */;
> Error: PLAN ERROR: Failure while materializing expression in constant
> expression evaluator [SPLIT('foo,bar,buz', ',')]. Errors:
> Error in expression at index -1. Error: Only ProjectRecordBatch could have
> complex writer function. You are using complex writer function split in a
> non-project operation!. Full expression: --UNKNOWN EXPRESSION--.{code}
> Note that
> {code:java}
> SELECT split(a,',') AS a FROM (SELECT 'foo,bar,buz' AS a) /* OK */;
> +-+
> | a |
> +-+
> | ["foo","bar","buz"] |
> +-+
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7322) Align cast boolean and schema boolean conversion
[
https://issues.apache.org/jira/browse/DRILL-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902982#comment-16902982
]
Denys Ordynskiy commented on DRILL-7322:
I found 3 another data types that have differences between the Drill internal
casting using schema and external casting by cast( as ) function.
*Float*
all_types.csvh
|float_col|varchar_float_col|
|340282356779733661637539395458142568447|340282356779733661637539395458142568447|
|-340282356779733661637539395458142568447|-340282356779733661637539395458142568447|
{noformat}
create schema (
float_col float not null,
varchar_float_col varchar not null
) for table dfs.tmp.`ischema/all_types`;
{noformat}
{noformat}
select
float_col,
sqlTypeOf(float_col) float_col_type,
cast(varchar_float_col as float) casted_float,
sqlTypeOf(cast(varchar_float_col as float)) casted_float_type,
varchar_float_col
from dfs.tmp.`ischema/all_types`;
{noformat}
Result:
|float_col|float_col_type|casted_float|casted_float_type|varchar_float_col|
|{color:red}*Infinity*{color}|FLOAT|{color:red}*3.4028235E38*{color}|FLOAT|340282356779733661637539395458142568447|
|{color:red}*-Infinity*{color}|FLOAT|{color:red}*-3.4028235E38*{color}|FLOAT|-340282356779733661637539395458142568447|
--
*Date*
all_types.csvh
|date_col|varchar_date_col|
|1-1-1|1-1-1|
{noformat}
create schema (
date_col date not null,
varchar_date_col varchar not null
) for table dfs.tmp.`ischema/all_types`;
{noformat}
{noformat}
select
date_col,
sqlTypeOf(date_col) date_col_type,
cast(varchar_date_col as date) casted_date_col,
sqlTypeOf(cast(varchar_date_col as date)) casted_date_col_type,
TO_DATE(varchar_date_col, '-MM-dd') to_date_col,
sqlTypeOf(TO_DATE(varchar_date_col, '-MM-dd')) to_date_col_type,
varchar_date_col
from dfs.tmp.`ischema/all_types`;
{noformat}
Result:
|date_col|date_col_type|casted_date_col|casted_date_col_type|to_date_col|to_date_col_type|varchar_date_col|
|{color:red}*0001-01-01*{color}|DATE|{color:red}*2001-01-01*{color}|DATE|{color:red}*0001-01-01*{color}|DATE|1-1-1|
--
*Time*
all_types.csvh
|time_col|varchar_time_col|
|01:01:001|01:01:001|
|01:001:001|01:001:001|
|001:001:001|001:001:001|
|1:1:1.|1:1:1.|
{noformat}
create schema (
time_col time not null,
varchar_time_col varchar not null
) for table dfs.tmp.`ischema/all_types`;
{noformat}
{noformat}
select
varchar_time_col,
time_col,
sqlTypeOf(time_col) time_col_type,
cast(varchar_time_col as time) casted_time_col,
sqlTypeOf(cast(varchar_time_col as time)) casted_time_col_type
from dfs.tmp.`ischema/all_types`;
{noformat}
|varchar_time_col|time_col|time_col_type|casted_time_col|casted_time_col_type|
|01:01:001|{color:red}*ERROR*{color}|{color:red}*ERROR*{color}|01:01:01|TIME|
|01:001:001|{color:red}*ERROR*{color}|{color:red}*ERROR*{color}|01:01:01|TIME|
|001:001:001|{color:red}*ERROR*{color}|{color:red}*ERROR*{color}|01:01:01|TIME|
|1:1:1.|{color:red}*ERROR*{color}|{color:red}*ERROR*{color}|01:01:01|TIME|
'01:01:001' - Error: DATA_READ ERROR: Invalid format:
"01:01:{color:red}*001*{color}" is malformed at "1"
'01:001:001'- Error: DATA_READ ERROR: Invalid format:
"01:{color:red}*001:001*{color}" is malformed at "1:001"
'001:001:001' - Error: DATA_READ ERROR: Invalid format:
"{color:red}*001:001:001*{color}" is malformed at "1:001:001"
'1:1:1.'- Error: DATA_READ ERROR: Invalid format:
"{color:red}*1:1:1.*{color}" is malformed at "."
Schema file reading and to_time() function throws an error if there are > 2
digits in the time unit. But cast( as time) works fine with '001' time literals.
> Align cast boolean and schema boolean conversion
>
>
> Key: DRILL-7322
> URL: https://issues.apache.org/jira/browse/DRILL-7322
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Denys Ordynskiy
>Priority: Major
>
> Information schema file allows converting any string to the boolean data type.
> But "case(.. as boolean)" statement throws an error:
> {color:#d04437}UserRemoteException : SYSTEM ERROR: IllegalArgumentException:
> Invalid value for boolean: a
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
> IllegalArgumentException: Invalid value for boolean: a{color}
> *Information Schema file should allow using the same range of boolean
> literals as cast statement.*
> *Steps to reproduce:*
> Upload text file all_types.csvh to the DFS /tmp/ischema/all_types:
> {noformat}
> boolean_col,boolean_col_for_cast
> true,true
> 1,1
> t,t
> y,y
> yes,yes
> on,on
> false,false
> 0,0
> f,f
> n,n
> no,no
> off,off
> a,a
> -,-
> !,!
> `,`
> 7,7
> @,@
> ^,^
> *,*
> {noformat}
> *Create schema:*
> {noformat}
> create
[jira] [Commented] (DRILL-7341) Vector reAlloc may fails after exchange.
[
https://issues.apache.org/jira/browse/DRILL-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902899#comment-16902899
]
ASF GitHub Bot commented on DRILL-7341:
---
oleg-zinovev commented on pull request #1838: DRILL-7341: Vector reAlloc may
fails after exchange
URL: https://github.com/apache/drill/pull/1838#discussion_r311989195
##
File path: exec/vector/src/main/codegen/templates/FixedValueVectors.java
##
@@ -210,10 +210,16 @@ public void reAlloc() {
// a zero-length buffer. Instead, just allocate a 256 byte
// buffer if we start at 0.
-final long newAllocationSize = allocationSizeInBytes == 0
+long newAllocationSize = allocationSizeInBytes == 0
? 256
: allocationSizeInBytes * 2L;
+// Some operations, such as Value Vector#exchange, can be change DrillBuf
data field without corresponding allocation size changes.
+// Check that the size of the allocation is sufficient to copy the old
buffer.
+while (newAllocationSize < data.capacity()) {
Review comment:
Done.
Could you clarify why DrillBuf#setZero performed only for FixedValueVectors?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Vector reAlloc may fails after exchange.
>
>
> Key: DRILL-7341
> URL: https://issues.apache.org/jira/browse/DRILL-7341
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: stacktrace.log
>
>
> There are several methods that modify the BaseDataValueVector#data field.
> Some of them, such as BaseDataValueVector#exchange, do not change
> allocationSizeInBytes.
> Therefore, if BaseDataValueVector#exchange was executed for vectors with
> different size, *ValueVector#reAlloc may create a buffer of insufficient size.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7341) Vector reAlloc may fails after exchange.
[
https://issues.apache.org/jira/browse/DRILL-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902895#comment-16902895
]
ASF GitHub Bot commented on DRILL-7341:
---
oleg-zinovev commented on pull request #1838: DRILL-7341: Vector reAlloc may
fails after exchange
URL: https://github.com/apache/drill/pull/1838#discussion_r311989195
##
File path: exec/vector/src/main/codegen/templates/FixedValueVectors.java
##
@@ -210,10 +210,16 @@ public void reAlloc() {
// a zero-length buffer. Instead, just allocate a 256 byte
// buffer if we start at 0.
-final long newAllocationSize = allocationSizeInBytes == 0
+long newAllocationSize = allocationSizeInBytes == 0
? 256
: allocationSizeInBytes * 2L;
+// Some operations, such as Value Vector#exchange, can be change DrillBuf
data field without corresponding allocation size changes.
+// Check that the size of the allocation is sufficient to copy the old
buffer.
+while (newAllocationSize < data.capacity()) {
Review comment:
Done.
Could you clarify why DrillBug#setZero performed only for FixedValueVectors?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Vector reAlloc may fails after exchange.
>
>
> Key: DRILL-7341
> URL: https://issues.apache.org/jira/browse/DRILL-7341
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.16.0
>Reporter: Oleg Zinoviev
>Priority: Major
> Attachments: stacktrace.log
>
>
> There are several methods that modify the BaseDataValueVector#data field.
> Some of them, such as BaseDataValueVector#exchange, do not change
> allocationSizeInBytes.
> Therefore, if BaseDataValueVector#exchange was executed for vectors with
> different size, *ValueVector#reAlloc may create a buffer of insufficient size.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7337) Add vararg UDFs support
[
https://issues.apache.org/jira/browse/DRILL-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902893#comment-16902893
]
ASF GitHub Bot commented on DRILL-7337:
---
vvysotskyi commented on pull request #1835: DRILL-7337: Add vararg UDFs support
URL: https://github.com/apache/drill/pull/1835
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add vararg UDFs support
> ---
>
> Key: DRILL-7337
> URL: https://issues.apache.org/jira/browse/DRILL-7337
> Project: Apache Drill
> Issue Type: Sub-task
>Affects Versions: 1.16.0
>Reporter: Volodymyr Vysotskyi
>Assignee: Volodymyr Vysotskyi
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.17.0
>
>
> The aim of this Jira is to add support for vararg UDFs to simplify UDFs
> creation for the case when it is required to accept different numbers of
> arguments.
> h2. Requirements for vararg UDFs:
> * It should be possible to register vararg UDFs with the same name, but with
> different argument types;
> * Only vararg UDFs with a single variable-length argument placed after all
> other arguments should be allowed;
> * Vararg UDF should have less priority than the regular one for the case
> when they both are suitable;
> * Besides simple functions, vararg support should be added to the aggregate
> functions.
> h2. Implementation details
> The lifecycle of UDF is the following:
> * UDF is validated in {{FunctionConverter}} class and for the case when
> there is no problem (UDF has required fields with required types, required
> annotations, etc.), it is converted to the {{DrillFuncHolder}} to be
> registered in the function registry. Also, corresponding {{SqlFunction}}
> instances are created based on {{DrillFuncHolder}} to be used in Calcite;
> * When a query uses this UDF, Calcite validate that UDF with required name,
> arguments number and arguments types (for Drill arguments types are not
> checked at this stage) exists;
> * After Calcite was able to find the required {{SqlFunction instance}}, it
> uses Drill to find required {{DrillFuncHolder}}. All the work for determining
> the most suitable function is done in {{FunctionResolver}} and in
> {{TypeCastRules.getCost()}};
> * At the execution stage, {{DrillFuncHolder}} found again using
> {{FunctionCall}} instance;
> * {{DrillFuncHolder}} is used for code generation.
> Considering these steps, the first thing to be done for adding support for
> vararg UDFs is updating logic in {{FunctionConverter}} to allow registering
> vararg UDFs taking into account requirements declared above.
> Calcite uses {{SqlOperandTypeChecker}} to verify arguments number, so Drill
> should provide its own for vararg UDFs to be able to use them. To determine
> whether UDF is vararg, new {{isVarArg}} property will be added to the
> {{FunctionTemplate}}.
> {{TypeCastRules.getCost()}} method should be updated to be able to find
> vararg UDFs and prioritize regular UDFs.
> Code generation logic should be updated to handle vararg UDFs. Generated code
> for varag argument will look in the following way:
> {code:java}
> NullableVarCharHolder[] inputs = new
> NullableVarCharHolder[3];
> inputs[0] = out14;
> inputs[1] = out19;
> inputs[2] = out24;
> {code}
> To create own varagr UDF, new {{isVarArg}} property should be set to {{true}}
> in {{FunctionTemplate}}.
> After that, required vararg input should be declared as an array.
> Here is an example if vararg UDF:
> {code:java}
> @FunctionTemplate(name = "concat_varchar",
> isVarArg = true,
> scope = FunctionTemplate.FunctionScope.SIMPLE)
> public class VarCharConcatFunction implements DrillSimpleFunc {
> @Param *VarCharHolder[] inputs*;
> @Output VarCharHolder out;
> @Inject DrillBuf buffer;
>
> @Override
> public void setup() {
> }
> @Override
> public void eval() {
> int length = 0;
> for (VarCharHolder input : inputs) {
> length += input.end - input.start;
> }
> out.buffer = buffer = buffer.reallocIfNeeded(length);
> out.start = out.end = 0;
> for (VarCharHolder input : inputs) {
> for (int id = input.start; id < input.end; id++) {
> out.buffer.setByte(out.end++, input.buffer.getByte(id));
> }
> }
> }
> }
> {code}
> h2. Limitations connected with VarArg UDFs:
> * Specified nulls handling in FunctionTemplate does not affect vararg
[jira] [Commented] (DRILL-7222) Visualize estimated and actual row counts for a query
[ https://issues.apache.org/jira/browse/DRILL-7222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902879#comment-16902879 ] ASF GitHub Bot commented on DRILL-7222: --- agozhiy commented on issue #1779: DRILL-7222: Visualize estimated and actual row counts for a query URL: https://github.com/apache/drill/pull/1779#issuecomment-519473200 @kkhatua, thank you for the changes, please answer just a few more comments I left in conversations. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Visualize estimated and actual row counts for a query > - > > Key: DRILL-7222 > URL: https://issues.apache.org/jira/browse/DRILL-7222 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.16.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Labels: doc-impacting, user-experience > Fix For: 1.17.0 > > > With statistics in place, it would be useful to have the *estimated* rowcount > along side the *actual* rowcount query profile's operator overview. > We can extract this from the Physical Plan section of the profile. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (DRILL-7222) Visualize estimated and actual row counts for a query
[
https://issues.apache.org/jira/browse/DRILL-7222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902873#comment-16902873
]
ASF GitHub Bot commented on DRILL-7222:
---
agozhiy commented on pull request #1779: DRILL-7222: Visualize estimated and
actual row counts for a query
URL: https://github.com/apache/drill/pull/1779#discussion_r311958036
##
File path: exec/java-exec/src/main/resources/rest/profile/profile.ftl
##
@@ -587,6 +622,49 @@
if (e.target.form)
<#if
model.isOnlyImpersonationEnabled()>doSubmitQueryWithUserName()<#else>doSubmitQueryWithAutoLimit();
});
+
+// Convert scientific to Decimal [Ref:
https://gist.github.com/jiggzson/b5f489af9ad931e3d186]
+function scientificToDecimal(num) {
Review comment:
From the Chrome web console:
`> parseInt(1.6E7)`
`< 1600`
So it works it seems.
Update: as you use Number(), this is not needed all the more.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Visualize estimated and actual row counts for a query
> -
>
> Key: DRILL-7222
> URL: https://issues.apache.org/jira/browse/DRILL-7222
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.16.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting, user-experience
> Fix For: 1.17.0
>
>
> With statistics in place, it would be useful to have the *estimated* rowcount
> along side the *actual* rowcount query profile's operator overview.
> We can extract this from the Physical Plan section of the profile.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7222) Visualize estimated and actual row counts for a query
[
https://issues.apache.org/jira/browse/DRILL-7222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902869#comment-16902869
]
ASF GitHub Bot commented on DRILL-7222:
---
agozhiy commented on pull request #1779: DRILL-7222: Visualize estimated and
actual row counts for a query
URL: https://github.com/apache/drill/pull/1779#discussion_r311966621
##
File path: exec/java-exec/src/main/resources/rest/profile/profile.ftl
##
@@ -84,6 +84,29 @@
document.getElementById(warningElemId).style.display="none";
}
+//Injects Estimated Rows
+function injectEstimatedRows() {
+ Object.keys(opRowCountMap).forEach(key => {
+var tgtElem = $("td.estRowsAnchor[key='"+key+"']");
+var status = tgtElem.append( "("+opRowCountMap[key]+")" );
Review comment:
Spaces are still present around the expression inside parentheses. This is
inconsistent with the overall code style.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Visualize estimated and actual row counts for a query
> -
>
> Key: DRILL-7222
> URL: https://issues.apache.org/jira/browse/DRILL-7222
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.16.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting, user-experience
> Fix For: 1.17.0
>
>
> With statistics in place, it would be useful to have the *estimated* rowcount
> along side the *actual* rowcount query profile's operator overview.
> We can extract this from the Physical Plan section of the profile.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (DRILL-7338) REST API calls to Drill fail due to insufficient heap memory
[
https://issues.apache.org/jira/browse/DRILL-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-7338:
Description:
Drill queries that use REST API calls have started failing(given below) after
recent changes.
{code:java}
RESOURCE ERROR: There is not enough heap memory to run this query using the web
interface.
Please try a query with fewer columns or with a filter or limit condition to
limit the data returned.
You can also try an ODBC/JDBC client.{code}
They were running fine earlier as the ResultSet returned was just few rows.
These queries now fail for even very small resultSets( < 10rows).
Investigating the issue revealed that we introduced a check to limit the Heap
usage.
The Wrapper code from
*_exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java_*
that throws this error, i see certain issues. It does seem we use a threshold
of *85%* of heap usage before throwing that warning and exiting the query.
{code:java}
public class QueryWrapper {
private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(QueryWrapper.class);
// Heap usage threshold/trigger to provide resiliency on web server for
queries submitted via HTTP
private static final double HEAP_MEMORY_FAILURE_THRESHOLD = 0.85;
...
private static MemoryMXBean memMXBean = ManagementFactory.getMemoryMXBean();
...
// Wait until the query execution is complete or there is error submitting
the query
logger.debug("Wait until the query execution is complete or there is error
submitting the query");
do {
try {
isComplete = webUserConnection.await(TimeUnit.SECONDS.toMillis(1));
//periodically timeout 1 sec to check heap
} catch (InterruptedException e) {}
usagePercent = getHeapUsage();
if (usagePercent > HEAP_MEMORY_FAILURE_THRESHOLD) {
nearlyOutOfHeapSpace = true;
}
} while (!isComplete && !nearlyOutOfHeapSpace);
{code}
By using above check, we unintentionally invited all those issues that happen
with Java’s Heap usage. JVM does try to make maximum usage of HEAP until Minor
or Major GC kicks in i.e GC kicks after there is no more space left in
heap(eden or young gen).
The workarounds i can think of in order to resolve this issue are:
# Remove this check altogether so we know why it is filling up Heap.
# Advise the users to stop using REST for querying data.(We did this already).
*But not all users may not be happy with this suggestion.* There could be few
dynamic applications(dashboard, monitoring etc).
# Make the threshold high enough so that GC kicks in much better.
If not above options, we have to tune the Heap sizes of drillbit. A quick fix
would be to increase the threshold from 85% to 100%(option-3 above).
*For documentation*
New Drill configuration property - drill.exec.http.memory.heap.failure.threshold
was:
Drill queries that use REST API calls have started failing(given below) after
recent changes.
{code:java}
RESOURCE ERROR: There is not enough heap memory to run this query using the web
interface.
Please try a query with fewer columns or with a filter or limit condition to
limit the data returned.
You can also try an ODBC/JDBC client.{code}
They were running fine earlier as the ResultSet returned was just few rows.
These queries now fail for even very small resultSets( < 10rows).
Investigating the issue revealed that we introduced a check to limit the Heap
usage.
The Wrapper code from
*_exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java_*
that throws this error, i see certain issues. It does seem we use a threshold
of *85%* of heap usage before throwing that warning and exiting the query.
{code:java}
public class QueryWrapper {
private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(QueryWrapper.class);
// Heap usage threshold/trigger to provide resiliency on web server for
queries submitted via HTTP
private static final double HEAP_MEMORY_FAILURE_THRESHOLD = 0.85;
...
private static MemoryMXBean memMXBean = ManagementFactory.getMemoryMXBean();
...
// Wait until the query execution is complete or there is error submitting
the query
logger.debug("Wait until the query execution is complete or there is error
submitting the query");
do {
try {
isComplete = webUserConnection.await(TimeUnit.SECONDS.toMillis(1));
//periodically timeout 1 sec to check heap
} catch (InterruptedException e) {}
usagePercent = getHeapUsage();
if (usagePercent > HEAP_MEMORY_FAILURE_THRESHOLD) {
nearlyOutOfHeapSpace = true;
}
} while (!isComplete && !nearlyOutOfHeapSpace);
{code}
By using above check, we unintentionally invited all those issues that happen
with Java’s Heap usage. JVM does try to make maximum usage of HEAP until Minor
or Major GC kicks in i.e GC kicks
[jira] [Updated] (DRILL-7338) REST API calls to Drill fail due to insufficient heap memory
[
https://issues.apache.org/jira/browse/DRILL-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-7338:
Labels: ready-to-commit (was: )
> REST API calls to Drill fail due to insufficient heap memory
>
>
> Key: DRILL-7338
> URL: https://issues.apache.org/jira/browse/DRILL-7338
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.15.0
>Reporter: Aditya Allamraju
>Assignee: Kunal Khatua
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.17.0
>
>
> Drill queries that use REST API calls have started failing(given below) after
> recent changes.
> {code:java}
> RESOURCE ERROR: There is not enough heap memory to run this query using the
> web interface.
> Please try a query with fewer columns or with a filter or limit condition to
> limit the data returned.
> You can also try an ODBC/JDBC client.{code}
> They were running fine earlier as the ResultSet returned was just few rows.
> These queries now fail for even very small resultSets( < 10rows).
> Investigating the issue revealed that we introduced a check to limit the Heap
> usage.
> The Wrapper code from
> *_exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java_*
> that throws this error, i see certain issues. It does seem we use a
> threshold of *85%* of heap usage before throwing that warning and exiting the
> query.
>
> {code:java}
> public class QueryWrapper {
> private static final org.slf4j.Logger logger =
> org.slf4j.LoggerFactory.getLogger(QueryWrapper.class);
> // Heap usage threshold/trigger to provide resiliency on web server for
> queries submitted via HTTP
> private static final double HEAP_MEMORY_FAILURE_THRESHOLD = 0.85;
> ...
> private static MemoryMXBean memMXBean = ManagementFactory.getMemoryMXBean();
> ...
> // Wait until the query execution is complete or there is error submitting
> the query
> logger.debug("Wait until the query execution is complete or there is
> error submitting the query");
> do {
> try {
> isComplete = webUserConnection.await(TimeUnit.SECONDS.toMillis(1));
> //periodically timeout 1 sec to check heap
> } catch (InterruptedException e) {}
> usagePercent = getHeapUsage();
> if (usagePercent > HEAP_MEMORY_FAILURE_THRESHOLD) {
> nearlyOutOfHeapSpace = true;
> }
> } while (!isComplete && !nearlyOutOfHeapSpace);
> {code}
> By using above check, we unintentionally invited all those issues that happen
> with Java’s Heap usage. JVM does try to make maximum usage of HEAP until
> Minor or Major GC kicks in i.e GC kicks after there is no more space left in
> heap(eden or young gen).
> The workarounds i can think of in order to resolve this issue are:
> # Remove this check altogether so we know why it is filling up Heap.
> # Advise the users to stop using REST for querying data.(We did this
> already). *But not all users may not be happy with this suggestion.* There
> could be few dynamic applications(dashboard, monitoring etc).
> # Make the threshold high enough so that GC kicks in much better.
> If not above options, we have to tune the Heap sizes of drillbit. A quick fix
> would be to increase the threshold from 85% to 100%(option-3 above).
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (DRILL-7338) REST API calls to Drill fail due to insufficient heap memory
[
https://issues.apache.org/jira/browse/DRILL-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-7338:
Labels: doc-impacting ready-to-commit (was: ready-to-commit)
> REST API calls to Drill fail due to insufficient heap memory
>
>
> Key: DRILL-7338
> URL: https://issues.apache.org/jira/browse/DRILL-7338
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.15.0
>Reporter: Aditya Allamraju
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.17.0
>
>
> Drill queries that use REST API calls have started failing(given below) after
> recent changes.
> {code:java}
> RESOURCE ERROR: There is not enough heap memory to run this query using the
> web interface.
> Please try a query with fewer columns or with a filter or limit condition to
> limit the data returned.
> You can also try an ODBC/JDBC client.{code}
> They were running fine earlier as the ResultSet returned was just few rows.
> These queries now fail for even very small resultSets( < 10rows).
> Investigating the issue revealed that we introduced a check to limit the Heap
> usage.
> The Wrapper code from
> *_exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java_*
> that throws this error, i see certain issues. It does seem we use a
> threshold of *85%* of heap usage before throwing that warning and exiting the
> query.
>
> {code:java}
> public class QueryWrapper {
> private static final org.slf4j.Logger logger =
> org.slf4j.LoggerFactory.getLogger(QueryWrapper.class);
> // Heap usage threshold/trigger to provide resiliency on web server for
> queries submitted via HTTP
> private static final double HEAP_MEMORY_FAILURE_THRESHOLD = 0.85;
> ...
> private static MemoryMXBean memMXBean = ManagementFactory.getMemoryMXBean();
> ...
> // Wait until the query execution is complete or there is error submitting
> the query
> logger.debug("Wait until the query execution is complete or there is
> error submitting the query");
> do {
> try {
> isComplete = webUserConnection.await(TimeUnit.SECONDS.toMillis(1));
> //periodically timeout 1 sec to check heap
> } catch (InterruptedException e) {}
> usagePercent = getHeapUsage();
> if (usagePercent > HEAP_MEMORY_FAILURE_THRESHOLD) {
> nearlyOutOfHeapSpace = true;
> }
> } while (!isComplete && !nearlyOutOfHeapSpace);
> {code}
> By using above check, we unintentionally invited all those issues that happen
> with Java’s Heap usage. JVM does try to make maximum usage of HEAP until
> Minor or Major GC kicks in i.e GC kicks after there is no more space left in
> heap(eden or young gen).
> The workarounds i can think of in order to resolve this issue are:
> # Remove this check altogether so we know why it is filling up Heap.
> # Advise the users to stop using REST for querying data.(We did this
> already). *But not all users may not be happy with this suggestion.* There
> could be few dynamic applications(dashboard, monitoring etc).
> # Make the threshold high enough so that GC kicks in much better.
> If not above options, we have to tune the Heap sizes of drillbit. A quick fix
> would be to increase the threshold from 85% to 100%(option-3 above).
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7338) REST API calls to Drill fail due to insufficient heap memory
[
https://issues.apache.org/jira/browse/DRILL-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902858#comment-16902858
]
ASF GitHub Bot commented on DRILL-7338:
---
arina-ielchiieva commented on issue #1837: DRILL-7338: REST API calls to Drill
fail due to insufficient heap memory
URL: https://github.com/apache/drill/pull/1837#issuecomment-519458040
+1, please squash the commits.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> REST API calls to Drill fail due to insufficient heap memory
>
>
> Key: DRILL-7338
> URL: https://issues.apache.org/jira/browse/DRILL-7338
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.15.0
>Reporter: Aditya Allamraju
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.17.0
>
>
> Drill queries that use REST API calls have started failing(given below) after
> recent changes.
> {code:java}
> RESOURCE ERROR: There is not enough heap memory to run this query using the
> web interface.
> Please try a query with fewer columns or with a filter or limit condition to
> limit the data returned.
> You can also try an ODBC/JDBC client.{code}
> They were running fine earlier as the ResultSet returned was just few rows.
> These queries now fail for even very small resultSets( < 10rows).
> Investigating the issue revealed that we introduced a check to limit the Heap
> usage.
> The Wrapper code from
> *_exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java_*
> that throws this error, i see certain issues. It does seem we use a
> threshold of *85%* of heap usage before throwing that warning and exiting the
> query.
>
> {code:java}
> public class QueryWrapper {
> private static final org.slf4j.Logger logger =
> org.slf4j.LoggerFactory.getLogger(QueryWrapper.class);
> // Heap usage threshold/trigger to provide resiliency on web server for
> queries submitted via HTTP
> private static final double HEAP_MEMORY_FAILURE_THRESHOLD = 0.85;
> ...
> private static MemoryMXBean memMXBean = ManagementFactory.getMemoryMXBean();
> ...
> // Wait until the query execution is complete or there is error submitting
> the query
> logger.debug("Wait until the query execution is complete or there is
> error submitting the query");
> do {
> try {
> isComplete = webUserConnection.await(TimeUnit.SECONDS.toMillis(1));
> //periodically timeout 1 sec to check heap
> } catch (InterruptedException e) {}
> usagePercent = getHeapUsage();
> if (usagePercent > HEAP_MEMORY_FAILURE_THRESHOLD) {
> nearlyOutOfHeapSpace = true;
> }
> } while (!isComplete && !nearlyOutOfHeapSpace);
> {code}
> By using above check, we unintentionally invited all those issues that happen
> with Java’s Heap usage. JVM does try to make maximum usage of HEAP until
> Minor or Major GC kicks in i.e GC kicks after there is no more space left in
> heap(eden or young gen).
> The workarounds i can think of in order to resolve this issue are:
> # Remove this check altogether so we know why it is filling up Heap.
> # Advise the users to stop using REST for querying data.(We did this
> already). *But not all users may not be happy with this suggestion.* There
> could be few dynamic applications(dashboard, monitoring etc).
> # Make the threshold high enough so that GC kicks in much better.
> If not above options, we have to tune the Heap sizes of drillbit. A quick fix
> would be to increase the threshold from 85% to 100%(option-3 above).
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7222) Visualize estimated and actual row counts for a query
[
https://issues.apache.org/jira/browse/DRILL-7222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902854#comment-16902854
]
ASF GitHub Bot commented on DRILL-7222:
---
agozhiy commented on pull request #1779: DRILL-7222: Visualize estimated and
actual row counts for a query
URL: https://github.com/apache/drill/pull/1779#discussion_r311958036
##
File path: exec/java-exec/src/main/resources/rest/profile/profile.ftl
##
@@ -587,6 +622,49 @@
if (e.target.form)
<#if
model.isOnlyImpersonationEnabled()>doSubmitQueryWithUserName()<#else>doSubmitQueryWithAutoLimit();
});
+
+// Convert scientific to Decimal [Ref:
https://gist.github.com/jiggzson/b5f489af9ad931e3d186]
+function scientificToDecimal(num) {
Review comment:
From the Chrome web console:
`> parseInt(1.6E7)`
`< 1600`
So it works it seems.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Visualize estimated and actual row counts for a query
> -
>
> Key: DRILL-7222
> URL: https://issues.apache.org/jira/browse/DRILL-7222
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.16.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting, user-experience
> Fix For: 1.17.0
>
>
> With statistics in place, it would be useful to have the *estimated* rowcount
> along side the *actual* rowcount query profile's operator overview.
> We can extract this from the Physical Plan section of the profile.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (DRILL-7222) Visualize estimated and actual row counts for a query
[
https://issues.apache.org/jira/browse/DRILL-7222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902846#comment-16902846
]
ASF GitHub Bot commented on DRILL-7222:
---
agozhiy commented on pull request #1779: DRILL-7222: Visualize estimated and
actual row counts for a query
URL: https://github.com/apache/drill/pull/1779#discussion_r311953200
##
File path: exec/java-exec/src/main/resources/rest/profile/profile.ftl
##
@@ -587,6 +622,49 @@
if (e.target.form)
<#if
model.isOnlyImpersonationEnabled()>doSubmitQueryWithUserName()<#else>doSubmitQueryWithAutoLimit();
});
+
+// Convert scientific to Decimal [Ref:
https://gist.github.com/jiggzson/b5f489af9ad931e3d186]
+function scientificToDecimal(num) {
+ //if the number is in scientific notation remove it
+ if(/\d+\.?\d*e[\+\-]*\d+/i.test(num)) {
+var zero = '0',
+parts = String(num).toLowerCase().split('e'), //split into coeff
and exponent
+e = parts.pop(),//store the exponential part
+l = Math.abs(e), //get the number of zeros
+sign = e/l,
+coeff_array = parts[0].split('.');
+if(sign === -1) {
+coeff_array[0] = Math.abs(coeff_array[0]);
+num = '-'+zero + '.' + new Array(l).join(zero) +
coeff_array.join('');
+}
+else {
+var dec = coeff_array[1];
+if(dec) l = l - dec.length;
+num = coeff_array.join('') + new Array(l+1).join(zero);
+}
+ }
+ return num;
+}
+
+// Extract estimated rowcount map
+var opRowCountMap = {};
+// Get OpId-Rowocunt Map
+function buildRowCountMap() {
+ var phyText = $('#query-physical').find('pre').text();
+ var opLines = phyText.split("\n");
+ for (var l in opLines) {
+var line = opLines[l];
+if (line.trim().length > 0) {
+ var opId = line.match(/\d+-\d+/g)[0];
+ var opRowCount = line.match(/rowcount = \S+/g)[0].split('
')[2].replace(',','').trim();
Review comment:
Right. Still, it can be simplified. This should work with both formats:
`line.match(/rowcount = ([^,]+)/)[1]`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Visualize estimated and actual row counts for a query
> -
>
> Key: DRILL-7222
> URL: https://issues.apache.org/jira/browse/DRILL-7222
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.16.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Labels: doc-impacting, user-experience
> Fix For: 1.17.0
>
>
> With statistics in place, it would be useful to have the *estimated* rowcount
> along side the *actual* rowcount query profile's operator overview.
> We can extract this from the Physical Plan section of the profile.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (DRILL-7321) split function doesn't work without from
[
https://issues.apache.org/jira/browse/DRILL-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Volodymyr Vysotskyi reassigned DRILL-7321:
--
Assignee: Volodymyr Vysotskyi
> split function doesn't work without from
>
>
> Key: DRILL-7321
> URL: https://issues.apache.org/jira/browse/DRILL-7321
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.16.0
>Reporter: benj
>Assignee: Volodymyr Vysotskyi
>Priority: Minor
>
> {code:java}
> SELECT upper('foo') AS a /* OK */;
> +-+
> | a |
> +-+
> | foo |
> +-+
> {code}
> but
> {code:java}
> SELECT split('foo,bar,buz',',') AS a /* NOK */;
> Error: PLAN ERROR: Failure while materializing expression in constant
> expression evaluator [SPLIT('foo,bar,buz', ',')]. Errors:
> Error in expression at index -1. Error: Only ProjectRecordBatch could have
> complex writer function. You are using complex writer function split in a
> non-project operation!. Full expression: --UNKNOWN EXPRESSION--.{code}
> Note that
> {code:java}
> SELECT split(a,',') AS a FROM (SELECT 'foo,bar,buz' AS a) /* OK */;
> +-+
> | a |
> +-+
> | ["foo","bar","buz"] |
> +-+
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
