[jira] [Commented] (IMPALA-10749) Add Runtime Filter Publish info to Node Lifecycle Event Timeline

2021-08-25 Thread Jira 


[ 
https://issues.apache.org/jira/browse/IMPALA-10749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17404275#comment-17404275
 ] 

Zoltán Borók-Nagy commented on IMPALA-10749:


Thanks for looking into it [~rizaon]. I think we only have info about 
first/last arrival time in the profile. If there's a complex query with lots of 
JOINs and runtime filters, then probably it'd be useful to know when the 
filters were published by the builders.

Though probably we can infer that information from other metrics of the 
builder/join with good enough accuracy, e.g. using "TotalTime" of the builder.

> Add Runtime Filter Publish info to Node Lifecycle Event Timeline
> 
>
> Key: IMPALA-10749
> URL: https://issues.apache.org/jira/browse/IMPALA-10749
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend
>Reporter: Zoltán Borók-Nagy
>Priority: Major
>
> Currently we only have the following info about runtime filters at the HASH 
> JOIN NODE in the profile:
> {noformat}
> Runtime filters: 1 of 1 Runtime Filter Published
> {noformat}
> But it would be useful to know when the runtime filters were published. We 
> could add this information to the Node Lifecycle Event Timeline.
> E.g. in test failures like IMPALA-10747 it would be good to know when the 
> runtime filters were published.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Updated] (IMPALA-10879) Add parquet stats to iceberg manifest

2021-08-25 Thread Jira 


 [ 
https://issues.apache.org/jira/browse/IMPALA-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltán Borók-Nagy updated IMPALA-10879:
---
Labels: impala-iceberg  (was: )

> Add parquet stats to iceberg manifest
> -
>
> Key: IMPALA-10879
> URL: https://issues.apache.org/jira/browse/IMPALA-10879
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend, Frontend
>Affects Versions: Impala 4.0.0
>Reporter: Attila Jeges
>Assignee: Attila Jeges
>Priority: Major
>  Labels: impala-iceberg
>
> Parquet stats should be written to iceberg manifest as per-datafile metrics.
> This task is specifically about the following metrics:
> - column_sizes : Map from column id to the total size on disk of all regions 
> that store the column. Does not include bytes necessary to read other 
> columns, like footers. Leave null for row-oriented formats
> - null_value_counts : Map from column id to number of null values in the 
> column.
> - lower_bounds : Map from column id to lower bound in the column serialized 
> as binary. Each value must be less than or equal to all non-null, non-NaN 
> values in the column for the file.
> - upper_bounds : Map from column id to upper bound in the column serialized 
> as binary. Each value must be greater than or equal to all non-null, non-Nan 
> values in the column for the file.
> Iceberg manifest doc: 
> https://iceberg.apache.org/spec/#manifests
> lower_bounds and upper_bounds values should be Single-value serialized to 
> binary:
> https://iceberg.apache.org/spec/#appendix-d-single-value-serialization



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Work started] (IMPALA-10879) Add parquet stats to iceberg manifest

2021-08-25 Thread Attila Jeges (Jira) 


 [ 
https://issues.apache.org/jira/browse/IMPALA-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on IMPALA-10879 started by Attila Jeges.
-
> Add parquet stats to iceberg manifest
> -
>
> Key: IMPALA-10879
> URL: https://issues.apache.org/jira/browse/IMPALA-10879
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend, Frontend
>Affects Versions: Impala 4.0.0
>Reporter: Attila Jeges
>Assignee: Attila Jeges
>Priority: Major
>  Labels: impala-iceberg
>
> Parquet stats should be written to iceberg manifest as per-datafile metrics.
> This task is specifically about the following metrics:
> - column_sizes : Map from column id to the total size on disk of all regions 
> that store the column. Does not include bytes necessary to read other 
> columns, like footers. Leave null for row-oriented formats
> - null_value_counts : Map from column id to number of null values in the 
> column.
> - lower_bounds : Map from column id to lower bound in the column serialized 
> as binary. Each value must be less than or equal to all non-null, non-NaN 
> values in the column for the file.
> - upper_bounds : Map from column id to upper bound in the column serialized 
> as binary. Each value must be greater than or equal to all non-null, non-Nan 
> values in the column for the file.
> Iceberg manifest doc: 
> https://iceberg.apache.org/spec/#manifests
> lower_bounds and upper_bounds values should be Single-value serialized to 
> binary:
> https://iceberg.apache.org/spec/#appendix-d-single-value-serialization



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-10879) Add parquet stats to iceberg manifest

2021-08-25 Thread Attila Jeges (Jira) 


[ 
https://issues.apache.org/jira/browse/IMPALA-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17404526#comment-17404526
 ] 

Attila Jeges commented on IMPALA-10879:
---

CR: https://gerrit.cloudera.org/#/c/17806/

> Add parquet stats to iceberg manifest
> -
>
> Key: IMPALA-10879
> URL: https://issues.apache.org/jira/browse/IMPALA-10879
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Backend, Frontend
>Affects Versions: Impala 4.0.0
>Reporter: Attila Jeges
>Assignee: Attila Jeges
>Priority: Major
>  Labels: impala-iceberg
>
> Parquet stats should be written to iceberg manifest as per-datafile metrics.
> This task is specifically about the following metrics:
> - column_sizes : Map from column id to the total size on disk of all regions 
> that store the column. Does not include bytes necessary to read other 
> columns, like footers. Leave null for row-oriented formats
> - null_value_counts : Map from column id to number of null values in the 
> column.
> - lower_bounds : Map from column id to lower bound in the column serialized 
> as binary. Each value must be less than or equal to all non-null, non-NaN 
> values in the column for the file.
> - upper_bounds : Map from column id to upper bound in the column serialized 
> as binary. Each value must be greater than or equal to all non-null, non-Nan 
> values in the column for the file.
> Iceberg manifest doc: 
> https://iceberg.apache.org/spec/#manifests
> lower_bounds and upper_bounds values should be Single-value serialized to 
> binary:
> https://iceberg.apache.org/spec/#appendix-d-single-value-serialization



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-10874) Upgrade impyla to the latest version

2021-08-25 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/IMPALA-10874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17404560#comment-17404560
 ] 

ASF subversion and git services commented on IMPALA-10874:
--

Commit 237ed5e8738ec565bc8d3ce813d9b70c12ad4ce7 in impala's branch 
refs/heads/master from wzhou-code
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=237ed5e ]

IMPALA-10874: Upgrade impyla to the latest version

This patch upgrades impyla to the latest version 0.18a1, which supports
cookie retention for LDAP authentications. Also adds unit-test cases
for implyla's HTTP test with LDAP authentication.

Testing:
 - Passed core tests.

Change-Id: I990e5cdde4e98d6ab3581fe48f53a5d0590ce492
Reviewed-on: http://gerrit.cloudera.org:8080/17795
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 


> Upgrade impyla to the latest version
> 
>
> Key: IMPALA-10874
> URL: https://issues.apache.org/jira/browse/IMPALA-10874
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Clients
>Affects Versions: Impala 4.1.0
>Reporter: Wenzhe Zhou
>Assignee: Wenzhe Zhou
>Priority: Major
> Fix For: Impala 4.1.0
>
>
> A new version 0.18a1 of impyla was published recently, which supports cookie 
> retention for LDAP authentications and jwt authentication. impyla was not 
> updated since 0.17a3. 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-10849) A LIKE predicate that ends in an escaped wildcard is incorrectly evaluated

2021-08-25 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/IMPALA-10849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17404558#comment-17404558
 ] 

ASF subversion and git services commented on IMPALA-10849:
--

Commit b54d0c35ffd354ee1ac9bc781a8ff36a64125676 in impala's branch 
refs/heads/master from Andrew Sherman
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=b54d0c3 ]

IMPALA-10849: Ignore escaped wildcards that terminate like predicates.

A like predicate is generally evaluated by converting it into a regex
that is evaluated at execution time. If the predicate of a like clause
is a constant (which is the common case when you say "row
like 'start%'") then there are optimizations where some cases that are
simpler then a regex are spotted, and a simple function than a regex
evaluator is used. One example is that a predicate such as ‘start%’ is
evaluated by looking for strings that begin with "start". Amusingly the
code that spots the potential optimizations uses regexes to look for
patterns in the like predicate. The code that looks for the
optimization where a simple prefix can be searched for does not deal
with the case where the '%' wildcard at the end of the predicate is
escaped. To fix this we add a test that deals with the case where the
predicate ends in an escaped '%'.

There are some other problems with escaped wildcards discussed in
IMPALA-2422. This change does not fix these problems, which are hard.

New tests for escaped wildcards are added to exprs.test - note that
these tests cannot be part of the LikeTbl tests as the like predicate
optimizations are only applied when the like predicate is a string
literal.

Exhaustive tests ran clean.

Change-Id: I30356c19f4f169d99f7cc6268937653af6b41b70
Reviewed-on: http://gerrit.cloudera.org:8080/17798
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 


> A LIKE predicate that ends in an escaped wildcard is incorrectly evaluated
> --
>
> Key: IMPALA-10849
> URL: https://issues.apache.org/jira/browse/IMPALA-10849
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>Priority: Critical
>
> If the last character of a LIKE predicate is an escaped wildcard (e.g. LIKE 
> foo\%) then it is incorrectly evaluated. This is because the fast path 
> optimizations in LikePrepareInternal treat the predicate as being a search 
> for a string with a fixed prefix. If the fast path optimizations are 
> commented out then the LIKE is evaluated correctly.
> A possible fix would be to make the fast path optimizations recognize that 
> escaped wildcards cannot be evaluated by the fixed prefix search.
> This is a simpler bug than that discussed in IMPALA-2422 which is to do with 
> ambiguities in the logic of unescaping string literals (which is more tricky 
> to fix).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Commented] (IMPALA-2422) % escaping does not work correctly in a LIKE clause

2021-08-25 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/IMPALA-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17404559#comment-17404559
 ] 

ASF subversion and git services commented on IMPALA-2422:
-

Commit b54d0c35ffd354ee1ac9bc781a8ff36a64125676 in impala's branch 
refs/heads/master from Andrew Sherman
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=b54d0c3 ]

IMPALA-10849: Ignore escaped wildcards that terminate like predicates.

A like predicate is generally evaluated by converting it into a regex
that is evaluated at execution time. If the predicate of a like clause
is a constant (which is the common case when you say "row
like 'start%'") then there are optimizations where some cases that are
simpler then a regex are spotted, and a simple function than a regex
evaluator is used. One example is that a predicate such as ‘start%’ is
evaluated by looking for strings that begin with "start". Amusingly the
code that spots the potential optimizations uses regexes to look for
patterns in the like predicate. The code that looks for the
optimization where a simple prefix can be searched for does not deal
with the case where the '%' wildcard at the end of the predicate is
escaped. To fix this we add a test that deals with the case where the
predicate ends in an escaped '%'.

There are some other problems with escaped wildcards discussed in
IMPALA-2422. This change does not fix these problems, which are hard.

New tests for escaped wildcards are added to exprs.test - note that
these tests cannot be part of the LikeTbl tests as the like predicate
optimizations are only applied when the like predicate is a string
literal.

Exhaustive tests ran clean.

Change-Id: I30356c19f4f169d99f7cc6268937653af6b41b70
Reviewed-on: http://gerrit.cloudera.org:8080/17798
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 


> % escaping does not work correctly in a LIKE clause
> ---
>
> Key: IMPALA-2422
> URL: https://issues.apache.org/jira/browse/IMPALA-2422
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend, Frontend
>Affects Versions: Impala 2.2.4, Impala 2.3.0, Impala 2.5.0, Impala 2.4.0, 
> Impala 2.6.0, Impala 2.7.0
>Reporter: Huaisi Xu
>Priority: Critical
>  Labels: correctness, downgraded, incompatibility
>
> {code:java}
> [localhost:21000] > select '%' like "\%";
> Query: select '%' like "\%"
> +---+
> | '%' like '\%' |
> +---+
> | false   |   -> should return true.
> +---+
> Fetched 1 row(s) in 0.01s
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Created] (IMPALA-10883) Dense runtime profile missing some fragment instance profile

2021-08-25 Thread Riza Suminto (Jira) 
Riza Suminto created IMPALA-10883:
-

 Summary: Dense runtime profile missing some fragment instance 
profile
 Key: IMPALA-10883
 URL: https://issues.apache.org/jira/browse/IMPALA-10883
 Project: IMPALA
  Issue Type: Bug
  Components: Distributed Exec
Affects Versions: Impala 4.0.0
Reporter: Riza Suminto


I ran TPC-DS Q78 in the following setup:
 * Cluster of 20 nodes
 * MT_DOP=12
 * --gen_experimental_profile=true

When I check the query profile of the completed query, I notice that couple 
fragment are missing profile from some instances. This missing profile is 
indicated in the missing instances id like this:
{code:java}
Fragment F00 [228 instances]:
 Instances: Instance 204c719f48777536:b83eb0ed0001 
(host=ia0306.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0002 (host=ia0306.halxg.cloudera.com:27000), , 
Instance 204c719f48777536:b83eb0ed0004 
(host=ia0306.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0005 (host=ia0306.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0006 
(host=ia0306.halxg.cloudera.com:27000), , , Instance 
204c719f48777536:b83eb0ed0009 (host=ia0306.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed000a 
(host=ia0306.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed000b (host=ia0306.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed000c 
(host=ia0306.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed000d (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed000e 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed000f (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0010 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0011 (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0012 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0013 (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0014 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0015 (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0016 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0017 (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0018 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0019 (host=ia0322.halxg.cloudera.com:27000), , , , 
Instance 204c719f48777536:b83eb0ed001d 
(host=ia0322.halxg.cloudera.com:27000), , , , Instance 
204c719f48777536:b83eb0ed0021 (host=ia0322.halxg.cloudera.com:27000), , , , 
Instance 204c719f48777536:b83eb0ed0025 
(host=ia0317.halxg.cloudera.com:27000), , , , ...{code}
Note several missing instances from host ia0306, ia0322, ia0317, and so on.

I tried to tune FLAGS_status_report_interval_ms faster to every 500ms and there 
are even more instance profile missing. On the other hand, setting 
FLAGS_status_report_interval_ms=0 (only send final report) will result in 
complete runtime profile.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Assigned] (IMPALA-10811) RPC to submit query getting stuck for AWS NLB forever.

2021-08-25 Thread Joe McDonnell (Jira) 


 [ 
https://issues.apache.org/jira/browse/IMPALA-10811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe McDonnell reassigned IMPALA-10811:
--

Assignee: Joe McDonnell

> RPC to submit query getting stuck for AWS NLB forever.
> --
>
> Key: IMPALA-10811
> URL: https://issues.apache.org/jira/browse/IMPALA-10811
> Project: IMPALA
>  Issue Type: Bug
>Reporter: Amogh Margoor
>Assignee: Joe McDonnell
>Priority: Major
> Attachments: profile+(13).txt
>
>
> Initial RPC to submit a query and fetch the query handle can take quite long 
> time to return as it can do various operations for planning and submission 
> that involve executing  Catalog Operations like Rename, Alter Table Recover 
> partition  that can take time on tables with many 
> partitions([https://github.com/apache/impala/blob/1231208da7104c832c13f272d1e5b8f554d29337/be/src/exec/catalog-op-executor.cc#L92]).
>  Attached is the profile of one such DDL query (with few fields hidden).
> These RPCs are: 
> 1. Beeswax:
> [https://github.com/apache/impala/blob/b28da054f3595bb92873433211438306fc22fbc7/be/src/service/impala-beeswax-server.cc#L57]
> 2. HS2:
> [https://github.com/apache/impala/blob/b28da054f3595bb92873433211438306fc22fbc7/be/src/service/impala-hs2-server.cc#L462]
>  
> One of the side effects of such RPC taking long time is that clients such as 
> impala-shell using AWS NLB can get stuck for ever. The reason is NLB tracks 
> and closes connections after 350s and cannot be configured. But after closing 
> the connection it doesn;t send TCP RST to the client. Only when client tries 
> to send data or packets NLB issues back TCP RST to indicate connection is not 
> alive. Documentation is here: 
> [https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html#connection-idle-timeout].
>  Hence the impala-shell waiting for RPC to return gets stuck indefinitely.
> Hence, we may need to evaluate techniques for RPCs to return query handle 
> after
>  # Creating Driver: 
> [https://github.com/apache/impala/blob/b28da054f3595bb92873433211438306fc22fbc7/be/src/service/impala-server.cc#L1150]
>  # Register Query: 
> [https://github.com/apache/impala/blob/b28da054f3595bb92873433211438306fc22fbc7/be/src/service/impala-server.cc#L1168]
>  and execute later parts of RPC asynchronously in different thread without 
> blocking the RPC. That way clients can get query handle and poll for it for 
> state and results.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Assigned] (IMPALA-10883) Dense runtime profile missing some fragment instance profile

2021-08-25 Thread Riza Suminto (Jira) 


 [ 
https://issues.apache.org/jira/browse/IMPALA-10883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Riza Suminto reassigned IMPALA-10883:
-

Assignee: Riza Suminto

> Dense runtime profile missing some fragment instance profile
> 
>
> Key: IMPALA-10883
> URL: https://issues.apache.org/jira/browse/IMPALA-10883
> Project: IMPALA
>  Issue Type: Bug
>  Components: Distributed Exec
>Affects Versions: Impala 4.0.0
>Reporter: Riza Suminto
>Assignee: Riza Suminto
>Priority: Major
>
> I ran TPC-DS Q78 in the following setup:
>  * Cluster of 20 nodes
>  * MT_DOP=12
>  * --gen_experimental_profile=true
> When I check the query profile of the completed query, I notice that couple 
> fragment are missing profile from some instances. This missing profile is 
> indicated in the missing instances id like this:
> {code:java}
> Fragment F00 [228 instances]:
>  Instances: Instance 204c719f48777536:b83eb0ed0001 
> (host=ia0306.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed0002 (host=ia0306.halxg.cloudera.com:27000), , 
> Instance 204c719f48777536:b83eb0ed0004 
> (host=ia0306.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed0005 (host=ia0306.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed0006 
> (host=ia0306.halxg.cloudera.com:27000), , , Instance 
> 204c719f48777536:b83eb0ed0009 (host=ia0306.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed000a 
> (host=ia0306.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed000b (host=ia0306.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed000c 
> (host=ia0306.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed000d (host=ia0318.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed000e 
> (host=ia0318.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed000f (host=ia0318.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed0010 
> (host=ia0318.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed0011 (host=ia0318.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed0012 
> (host=ia0318.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed0013 (host=ia0318.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed0014 
> (host=ia0318.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed0015 (host=ia0318.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed0016 
> (host=ia0318.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed0017 (host=ia0318.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed0018 
> (host=ia0318.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed0019 (host=ia0322.halxg.cloudera.com:27000), , , 
> , Instance 204c719f48777536:b83eb0ed001d 
> (host=ia0322.halxg.cloudera.com:27000), , , , Instance 
> 204c719f48777536:b83eb0ed0021 (host=ia0322.halxg.cloudera.com:27000), , , 
> , Instance 204c719f48777536:b83eb0ed0025 
> (host=ia0317.halxg.cloudera.com:27000), , , , ...{code}
> Note several missing instances from host ia0306, ia0322, ia0317, and so on.
> I tried to tune FLAGS_status_report_interval_ms faster to every 500ms and 
> there are even more instance profile missing. On the other hand, setting 
> FLAGS_status_report_interval_ms=0 (only send final report) will result in 
> complete runtime profile.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Created] (IMPALA-10884) Improve pretty-printing of fragment instance name

2021-08-25 Thread Riza Suminto (Jira) 
Riza Suminto created IMPALA-10884:
-

 Summary: Improve pretty-printing of fragment instance name
 Key: IMPALA-10884
 URL: https://issues.apache.org/jira/browse/IMPALA-10884
 Project: IMPALA
  Issue Type: Improvement
  Components: Backend
Affects Versions: Impala 4.0.0
Reporter: Riza Suminto
Assignee: Riza Suminto


When using impala-profile-tool with extended verbosity, SummaryStatsCounter and 
TimeSeriesCounter coming from different instances are printed with prefix 
number like this:

 
{code:java}
- MemoryUsage[0] (2s000ms): 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 
57.26 MB, 62.01 MB, 63.80 MB, 63.80 MB, 63.80 MB, 63.78 MB, 63.08 MB, 61.77 MB, 
60.47 MB, 60.44 MB, 60.62 MB, 61.08 MB, 61.32 MB, 61.21 MB, 61.32 MB, 61.28 MB, 
61.45 MB, 61.35 MB, 61.30 MB, 61.32 MB, 61.43 MB, 61.28 MB, 61.56 MB, 61.49 MB, 
61.47 MB, 61.50 MB, 61.47 MB, 61.64 MB, 61.49 MB, 61.54 MB, 61.42 MB, 61.47 MB, 
61.51 MB
 - MemoryUsage[1] (2s000ms): 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 32.64 MB, 
57.42 MB, 63.74 MB, 63.74 MB, 63.74 MB, 63.74 MB, 63.89 MB, 63.41 MB, 61.98 MB, 
62.72 MB, 61.88 MB, 61.24 MB, 61.09 MB, 61.15 MB, 61.17 MB, 61.47 MB, 61.63 MB, 
61.68 MB, 61.38 MB, 61.57 MB, 61.58 MB, 61.44 MB, 61.61 MB, 61.62 MB, 61.37 MB, 
61.64 MB, 61.63 MB, 61.36 MB, 61.70 MB, 61.49 MB, 61.45 MB, 61.48 MB, 61.50 MB, 
61.44 MB
 - MemoryUsage[3] (2s000ms): 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 
39.72 MB, 45.50 MB, 49.18 MB, 52.27 MB, 52.27 MB, 51.80 MB, 49.63 MB, 49.91 MB, 
48.78 MB, 48.87 MB, 48.72 MB, 49.47 MB, 49.40 MB, 49.01 MB, 61.53 MB, 61.35 MB, 
61.29 MB, 61.27 MB, 61.43 MB, 61.58 MB, 61.59 MB, 61.58 MB, 61.54 MB, 61.58 MB, 
61.40 MB, 61.44 MB, 61.47 MB, 61.50 MB, 61.53 MB, 61.62 MB, 61.15 MB, 61.63 MB, 
61.62 MB
...
{code}
 

The prefix number maps into fragment instance from where that counter is coming 
from. However, the fragment instance name is still printed in single long line 
without prefix number like this:
Fragment F00 [228 instances]:
 Instances: Instance 204c719f48777536:b83eb0ed0001 
(host=ia0306.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0002 (host=ia0306.halxg.cloudera.com:27000), , 
Instance 204c719f48777536:b83eb0ed0004 
(host=ia0306.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0005 (host=ia0306.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0006 
(host=ia0306.halxg.cloudera.com:27000), , , Instance 
204c719f48777536:b83eb0ed0009 (host=ia0306.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed000a 
(host=ia0306.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed000b (host=ia0306.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed000c 
(host=ia0306.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed000d (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed000e 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed000f (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0010 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0011 (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0012 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0013 (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0014 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0015 (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0016 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0017 (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0018 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0019 (host=ia0322.halxg.cloudera.com:27000), , , , 
Instance 204c719f48777536:b83eb0ed001d 
(host=ia0322.halxg.cloudera.com:27000), , , , Instance 
204c719f48777536:b83eb0ed0021 (host=ia0322.halxg.cloudera.com:27000), , , , 
Instance 204c719f48777536:b83eb0ed0025 
(host=ia0317.halxg.cloudera.com:27000), , , , ...
This makes it hard to match counters with their fragment instance. We should 
break thing long line into multiple line and add prefix number.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org

[jira] [Updated] (IMPALA-10884) Improve pretty-printing of fragment instance name

2021-08-25 Thread Riza Suminto (Jira) 


 [ 
https://issues.apache.org/jira/browse/IMPALA-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Riza Suminto updated IMPALA-10884:
--
Description: 
When using impala-profile-tool with extended verbosity, SummaryStatsCounter and 
TimeSeriesCounter coming from different instances are printed with prefix 
number like this:

 
{code:java}
- MemoryUsage[0] (2s000ms): 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 
57.26 MB, 62.01 MB, 63.80 MB, 63.80 MB, 63.80 MB, 63.78 MB, 63.08 MB, 61.77 MB, 
60.47 MB, 60.44 MB, 60.62 MB, 61.08 MB, 61.32 MB, 61.21 MB, 61.32 MB, 61.28 MB, 
61.45 MB, 61.35 MB, 61.30 MB, 61.32 MB, 61.43 MB, 61.28 MB, 61.56 MB, 61.49 MB, 
61.47 MB, 61.50 MB, 61.47 MB, 61.64 MB, 61.49 MB, 61.54 MB, 61.42 MB, 61.47 MB, 
61.51 MB
 - MemoryUsage[1] (2s000ms): 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 32.64 MB, 
57.42 MB, 63.74 MB, 63.74 MB, 63.74 MB, 63.74 MB, 63.89 MB, 63.41 MB, 61.98 MB, 
62.72 MB, 61.88 MB, 61.24 MB, 61.09 MB, 61.15 MB, 61.17 MB, 61.47 MB, 61.63 MB, 
61.68 MB, 61.38 MB, 61.57 MB, 61.58 MB, 61.44 MB, 61.61 MB, 61.62 MB, 61.37 MB, 
61.64 MB, 61.63 MB, 61.36 MB, 61.70 MB, 61.49 MB, 61.45 MB, 61.48 MB, 61.50 MB, 
61.44 MB
 - MemoryUsage[3] (2s000ms): 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 
39.72 MB, 45.50 MB, 49.18 MB, 52.27 MB, 52.27 MB, 51.80 MB, 49.63 MB, 49.91 MB, 
48.78 MB, 48.87 MB, 48.72 MB, 49.47 MB, 49.40 MB, 49.01 MB, 61.53 MB, 61.35 MB, 
61.29 MB, 61.27 MB, 61.43 MB, 61.58 MB, 61.59 MB, 61.58 MB, 61.54 MB, 61.58 MB, 
61.40 MB, 61.44 MB, 61.47 MB, 61.50 MB, 61.53 MB, 61.62 MB, 61.15 MB, 61.63 MB, 
61.62 MB
...
{code}
 

The prefix number maps into fragment instance from where that counter is coming 
from. However, the fragment instance name is still printed in single long line 
without prefix number like this:
{code:java}
Fragment F00 [228 instances]:
 Instances: Instance 204c719f48777536:b83eb0ed0001 
(host=ia0306.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0002 (host=ia0306.halxg.cloudera.com:27000), , 
Instance 204c719f48777536:b83eb0ed0004 
(host=ia0306.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0005 (host=ia0306.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0006 
(host=ia0306.halxg.cloudera.com:27000), , , Instance 
204c719f48777536:b83eb0ed0009 (host=ia0306.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed000a 
(host=ia0306.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed000b (host=ia0306.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed000c 
(host=ia0306.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed000d (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed000e 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed000f (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0010 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0011 (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0012 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0013 (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0014 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0015 (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0016 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0017 (host=ia0318.halxg.cloudera.com:27000), 
Instance 204c719f48777536:b83eb0ed0018 
(host=ia0318.halxg.cloudera.com:27000), Instance 
204c719f48777536:b83eb0ed0019 (host=ia0322.halxg.cloudera.com:27000), , , , 
Instance 204c719f48777536:b83eb0ed001d 
(host=ia0322.halxg.cloudera.com:27000), , , , Instance 
204c719f48777536:b83eb0ed0021 (host=ia0322.halxg.cloudera.com:27000), , , , 
Instance 204c719f48777536:b83eb0ed0025 
(host=ia0317.halxg.cloudera.com:27000), , , , ...{code}

 This makes it hard to match counters with their fragment instance. We should 
break thing long line into multiple line and add prefix number.

  was:
When using impala-profile-tool with extended verbosity, SummaryStatsCounter and 
TimeSeriesCounter coming from different instances are printed with prefix 
number like this:

 
{code:java}
- MemoryUsage[0] (2s000ms): 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 
57.26 MB, 62.01 MB, 63.80 MB, 63.80 MB, 63.80 MB, 63.78 MB, 63.08 MB, 61.77 MB, 
60.47 MB, 60.44 MB, 60.62 MB, 61.08 MB, 61.32 MB, 61.21 MB, 61.32 MB, 61.28 MB, 
61.45 MB, 61.35 MB, 61.30 MB, 61.32 MB, 61.43 MB, 61.28 MB, 61.56 MB, 61.49 MB, 
61.47 MB, 61.50 MB, 61.47 MB, 61.64 MB, 61.49 MB, 61.54 MB, 61.42 MB, 61.47 MB, 
61.51 MB
 - MemoryUsage[1] (2s000ms): 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 32.64 MB, 
57.42 MB, 63.74 MB, 63.74 MB, 63.74 MB, 63.74 MB, 63.89 MB, 63.41 MB, 61.98 MB, 
62.72 MB, 61.88 MB, 61.24

[jira] [Commented] (IMPALA-10883) Dense runtime profile missing some fragment instance profile

2021-08-25 Thread Riza Suminto (Jira) 


[ 
https://issues.apache.org/jira/browse/IMPALA-10883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17404982#comment-17404982
 ] 

Riza Suminto commented on IMPALA-10883:
---

There seem to be different expectations between coordinator and reporter 
backend. Let say MT_DOP=12 and a backend sending runtime profile update several 
times.

Coordinator update its backend state profile through function call 
BackendState::ApplyExecStatusReport() -> 
AggregatedRuntimeProfile::UpdateAggregatedFromInstances().
UpdateAggregatedFromInstances() expects that the incoming TRuntimeProfileTree 
contains all 12 runtime profiles from 12 instances that run at the reporting 
backend.
[https://github.com/apache/impala/blob/237ed5e/be/src/util/runtime-profile.cc#L542-L544]

However, on the sender side, the backend can send a partial update. That is 
when some of the fragment instances finished earlier and already send their 
final status update.
The agg_profile is initialized with 12 runtime profiles slots. But the slot 
that belongs to the finished fragment instance will not be populated.
[https://github.com/apache/impala/blob/237ed5e/be/src/runtime/query-state.cc#L558-L575]

> Dense runtime profile missing some fragment instance profile
> 
>
> Key: IMPALA-10883
> URL: https://issues.apache.org/jira/browse/IMPALA-10883
> Project: IMPALA
>  Issue Type: Bug
>  Components: Distributed Exec
>Affects Versions: Impala 4.0.0
>Reporter: Riza Suminto
>Assignee: Riza Suminto
>Priority: Major
>
> I ran TPC-DS Q78 in the following setup:
>  * Cluster of 20 nodes
>  * MT_DOP=12
>  * --gen_experimental_profile=true
> When I check the query profile of the completed query, I notice that couple 
> fragment are missing profile from some instances. This missing profile is 
> indicated in the missing instances id like this:
> {code:java}
> Fragment F00 [228 instances]:
>  Instances: Instance 204c719f48777536:b83eb0ed0001 
> (host=ia0306.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed0002 (host=ia0306.halxg.cloudera.com:27000), , 
> Instance 204c719f48777536:b83eb0ed0004 
> (host=ia0306.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed0005 (host=ia0306.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed0006 
> (host=ia0306.halxg.cloudera.com:27000), , , Instance 
> 204c719f48777536:b83eb0ed0009 (host=ia0306.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed000a 
> (host=ia0306.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed000b (host=ia0306.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed000c 
> (host=ia0306.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed000d (host=ia0318.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed000e 
> (host=ia0318.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed000f (host=ia0318.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed0010 
> (host=ia0318.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed0011 (host=ia0318.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed0012 
> (host=ia0318.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed0013 (host=ia0318.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed0014 
> (host=ia0318.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed0015 (host=ia0318.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed0016 
> (host=ia0318.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed0017 (host=ia0318.halxg.cloudera.com:27000), 
> Instance 204c719f48777536:b83eb0ed0018 
> (host=ia0318.halxg.cloudera.com:27000), Instance 
> 204c719f48777536:b83eb0ed0019 (host=ia0322.halxg.cloudera.com:27000), , , 
> , Instance 204c719f48777536:b83eb0ed001d 
> (host=ia0322.halxg.cloudera.com:27000), , , , Instance 
> 204c719f48777536:b83eb0ed0021 (host=ia0322.halxg.cloudera.com:27000), , , 
> , Instance 204c719f48777536:b83eb0ed0025 
> (host=ia0317.halxg.cloudera.com:27000), , , , ...{code}
> Note several missing instances from host ia0306, ia0322, ia0317, and so on.
> I tried to tune FLAGS_status_report_interval_ms faster to every 500ms and 
> there are even more instance profile missing. On the other hand, setting 
> FLAGS_status_report_interval_ms=0 (only send final report) will result in 
> complete runtime profile.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org