[jira] [Commented] (IMPALA-10749) Add Runtime Filter Publish info to Node Lifecycle Event Timeline
[ https://issues.apache.org/jira/browse/IMPALA-10749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17404275#comment-17404275 ] Zoltán Borók-Nagy commented on IMPALA-10749: Thanks for looking into it [~rizaon]. I think we only have info about first/last arrival time in the profile. If there's a complex query with lots of JOINs and runtime filters, then probably it'd be useful to know when the filters were published by the builders. Though probably we can infer that information from other metrics of the builder/join with good enough accuracy, e.g. using "TotalTime" of the builder. > Add Runtime Filter Publish info to Node Lifecycle Event Timeline > > > Key: IMPALA-10749 > URL: https://issues.apache.org/jira/browse/IMPALA-10749 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Zoltán Borók-Nagy >Priority: Major > > Currently we only have the following info about runtime filters at the HASH > JOIN NODE in the profile: > {noformat} > Runtime filters: 1 of 1 Runtime Filter Published > {noformat} > But it would be useful to know when the runtime filters were published. We > could add this information to the Node Lifecycle Event Timeline. > E.g. in test failures like IMPALA-10747 it would be good to know when the > runtime filters were published. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10879) Add parquet stats to iceberg manifest
[ https://issues.apache.org/jira/browse/IMPALA-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltán Borók-Nagy updated IMPALA-10879: --- Labels: impala-iceberg (was: ) > Add parquet stats to iceberg manifest > - > > Key: IMPALA-10879 > URL: https://issues.apache.org/jira/browse/IMPALA-10879 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Affects Versions: Impala 4.0.0 >Reporter: Attila Jeges >Assignee: Attila Jeges >Priority: Major > Labels: impala-iceberg > > Parquet stats should be written to iceberg manifest as per-datafile metrics. > This task is specifically about the following metrics: > - column_sizes : Map from column id to the total size on disk of all regions > that store the column. Does not include bytes necessary to read other > columns, like footers. Leave null for row-oriented formats > - null_value_counts : Map from column id to number of null values in the > column. > - lower_bounds : Map from column id to lower bound in the column serialized > as binary. Each value must be less than or equal to all non-null, non-NaN > values in the column for the file. > - upper_bounds : Map from column id to upper bound in the column serialized > as binary. Each value must be greater than or equal to all non-null, non-Nan > values in the column for the file. > Iceberg manifest doc: > https://iceberg.apache.org/spec/#manifests > lower_bounds and upper_bounds values should be Single-value serialized to > binary: > https://iceberg.apache.org/spec/#appendix-d-single-value-serialization -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-10879) Add parquet stats to iceberg manifest
[ https://issues.apache.org/jira/browse/IMPALA-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-10879 started by Attila Jeges. - > Add parquet stats to iceberg manifest > - > > Key: IMPALA-10879 > URL: https://issues.apache.org/jira/browse/IMPALA-10879 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Affects Versions: Impala 4.0.0 >Reporter: Attila Jeges >Assignee: Attila Jeges >Priority: Major > Labels: impala-iceberg > > Parquet stats should be written to iceberg manifest as per-datafile metrics. > This task is specifically about the following metrics: > - column_sizes : Map from column id to the total size on disk of all regions > that store the column. Does not include bytes necessary to read other > columns, like footers. Leave null for row-oriented formats > - null_value_counts : Map from column id to number of null values in the > column. > - lower_bounds : Map from column id to lower bound in the column serialized > as binary. Each value must be less than or equal to all non-null, non-NaN > values in the column for the file. > - upper_bounds : Map from column id to upper bound in the column serialized > as binary. Each value must be greater than or equal to all non-null, non-Nan > values in the column for the file. > Iceberg manifest doc: > https://iceberg.apache.org/spec/#manifests > lower_bounds and upper_bounds values should be Single-value serialized to > binary: > https://iceberg.apache.org/spec/#appendix-d-single-value-serialization -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10879) Add parquet stats to iceberg manifest
[ https://issues.apache.org/jira/browse/IMPALA-10879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17404526#comment-17404526 ] Attila Jeges commented on IMPALA-10879: --- CR: https://gerrit.cloudera.org/#/c/17806/ > Add parquet stats to iceberg manifest > - > > Key: IMPALA-10879 > URL: https://issues.apache.org/jira/browse/IMPALA-10879 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Affects Versions: Impala 4.0.0 >Reporter: Attila Jeges >Assignee: Attila Jeges >Priority: Major > Labels: impala-iceberg > > Parquet stats should be written to iceberg manifest as per-datafile metrics. > This task is specifically about the following metrics: > - column_sizes : Map from column id to the total size on disk of all regions > that store the column. Does not include bytes necessary to read other > columns, like footers. Leave null for row-oriented formats > - null_value_counts : Map from column id to number of null values in the > column. > - lower_bounds : Map from column id to lower bound in the column serialized > as binary. Each value must be less than or equal to all non-null, non-NaN > values in the column for the file. > - upper_bounds : Map from column id to upper bound in the column serialized > as binary. Each value must be greater than or equal to all non-null, non-Nan > values in the column for the file. > Iceberg manifest doc: > https://iceberg.apache.org/spec/#manifests > lower_bounds and upper_bounds values should be Single-value serialized to > binary: > https://iceberg.apache.org/spec/#appendix-d-single-value-serialization -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10874) Upgrade impyla to the latest version
[ https://issues.apache.org/jira/browse/IMPALA-10874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17404560#comment-17404560 ] ASF subversion and git services commented on IMPALA-10874: -- Commit 237ed5e8738ec565bc8d3ce813d9b70c12ad4ce7 in impala's branch refs/heads/master from wzhou-code [ https://gitbox.apache.org/repos/asf?p=impala.git;h=237ed5e ] IMPALA-10874: Upgrade impyla to the latest version This patch upgrades impyla to the latest version 0.18a1, which supports cookie retention for LDAP authentications. Also adds unit-test cases for implyla's HTTP test with LDAP authentication. Testing: - Passed core tests. Change-Id: I990e5cdde4e98d6ab3581fe48f53a5d0590ce492 Reviewed-on: http://gerrit.cloudera.org:8080/17795 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Upgrade impyla to the latest version > > > Key: IMPALA-10874 > URL: https://issues.apache.org/jira/browse/IMPALA-10874 > Project: IMPALA > Issue Type: Improvement > Components: Clients >Affects Versions: Impala 4.1.0 >Reporter: Wenzhe Zhou >Assignee: Wenzhe Zhou >Priority: Major > Fix For: Impala 4.1.0 > > > A new version 0.18a1 of impyla was published recently, which supports cookie > retention for LDAP authentications and jwt authentication. impyla was not > updated since 0.17a3. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-10849) A LIKE predicate that ends in an escaped wildcard is incorrectly evaluated
[ https://issues.apache.org/jira/browse/IMPALA-10849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17404558#comment-17404558 ] ASF subversion and git services commented on IMPALA-10849: -- Commit b54d0c35ffd354ee1ac9bc781a8ff36a64125676 in impala's branch refs/heads/master from Andrew Sherman [ https://gitbox.apache.org/repos/asf?p=impala.git;h=b54d0c3 ] IMPALA-10849: Ignore escaped wildcards that terminate like predicates. A like predicate is generally evaluated by converting it into a regex that is evaluated at execution time. If the predicate of a like clause is a constant (which is the common case when you say "row like 'start%'") then there are optimizations where some cases that are simpler then a regex are spotted, and a simple function than a regex evaluator is used. One example is that a predicate such as ‘start%’ is evaluated by looking for strings that begin with "start". Amusingly the code that spots the potential optimizations uses regexes to look for patterns in the like predicate. The code that looks for the optimization where a simple prefix can be searched for does not deal with the case where the '%' wildcard at the end of the predicate is escaped. To fix this we add a test that deals with the case where the predicate ends in an escaped '%'. There are some other problems with escaped wildcards discussed in IMPALA-2422. This change does not fix these problems, which are hard. New tests for escaped wildcards are added to exprs.test - note that these tests cannot be part of the LikeTbl tests as the like predicate optimizations are only applied when the like predicate is a string literal. Exhaustive tests ran clean. Change-Id: I30356c19f4f169d99f7cc6268937653af6b41b70 Reviewed-on: http://gerrit.cloudera.org:8080/17798 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > A LIKE predicate that ends in an escaped wildcard is incorrectly evaluated > -- > > Key: IMPALA-10849 > URL: https://issues.apache.org/jira/browse/IMPALA-10849 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Critical > > If the last character of a LIKE predicate is an escaped wildcard (e.g. LIKE > foo\%) then it is incorrectly evaluated. This is because the fast path > optimizations in LikePrepareInternal treat the predicate as being a search > for a string with a fixed prefix. If the fast path optimizations are > commented out then the LIKE is evaluated correctly. > A possible fix would be to make the fast path optimizations recognize that > escaped wildcards cannot be evaluated by the fixed prefix search. > This is a simpler bug than that discussed in IMPALA-2422 which is to do with > ambiguities in the logic of unescaping string literals (which is more tricky > to fix). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-2422) % escaping does not work correctly in a LIKE clause
[ https://issues.apache.org/jira/browse/IMPALA-2422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17404559#comment-17404559 ] ASF subversion and git services commented on IMPALA-2422: - Commit b54d0c35ffd354ee1ac9bc781a8ff36a64125676 in impala's branch refs/heads/master from Andrew Sherman [ https://gitbox.apache.org/repos/asf?p=impala.git;h=b54d0c3 ] IMPALA-10849: Ignore escaped wildcards that terminate like predicates. A like predicate is generally evaluated by converting it into a regex that is evaluated at execution time. If the predicate of a like clause is a constant (which is the common case when you say "row like 'start%'") then there are optimizations where some cases that are simpler then a regex are spotted, and a simple function than a regex evaluator is used. One example is that a predicate such as ‘start%’ is evaluated by looking for strings that begin with "start". Amusingly the code that spots the potential optimizations uses regexes to look for patterns in the like predicate. The code that looks for the optimization where a simple prefix can be searched for does not deal with the case where the '%' wildcard at the end of the predicate is escaped. To fix this we add a test that deals with the case where the predicate ends in an escaped '%'. There are some other problems with escaped wildcards discussed in IMPALA-2422. This change does not fix these problems, which are hard. New tests for escaped wildcards are added to exprs.test - note that these tests cannot be part of the LikeTbl tests as the like predicate optimizations are only applied when the like predicate is a string literal. Exhaustive tests ran clean. Change-Id: I30356c19f4f169d99f7cc6268937653af6b41b70 Reviewed-on: http://gerrit.cloudera.org:8080/17798 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > % escaping does not work correctly in a LIKE clause > --- > > Key: IMPALA-2422 > URL: https://issues.apache.org/jira/browse/IMPALA-2422 > Project: IMPALA > Issue Type: Bug > Components: Backend, Frontend >Affects Versions: Impala 2.2.4, Impala 2.3.0, Impala 2.5.0, Impala 2.4.0, > Impala 2.6.0, Impala 2.7.0 >Reporter: Huaisi Xu >Priority: Critical > Labels: correctness, downgraded, incompatibility > > {code:java} > [localhost:21000] > select '%' like "\%"; > Query: select '%' like "\%" > +---+ > | '%' like '\%' | > +---+ > | false | -> should return true. > +---+ > Fetched 1 row(s) in 0.01s > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10883) Dense runtime profile missing some fragment instance profile
Riza Suminto created IMPALA-10883: - Summary: Dense runtime profile missing some fragment instance profile Key: IMPALA-10883 URL: https://issues.apache.org/jira/browse/IMPALA-10883 Project: IMPALA Issue Type: Bug Components: Distributed Exec Affects Versions: Impala 4.0.0 Reporter: Riza Suminto I ran TPC-DS Q78 in the following setup: * Cluster of 20 nodes * MT_DOP=12 * --gen_experimental_profile=true When I check the query profile of the completed query, I notice that couple fragment are missing profile from some instances. This missing profile is indicated in the missing instances id like this: {code:java} Fragment F00 [228 instances]: Instances: Instance 204c719f48777536:b83eb0ed0001 (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0002 (host=ia0306.halxg.cloudera.com:27000), , Instance 204c719f48777536:b83eb0ed0004 (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0005 (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0006 (host=ia0306.halxg.cloudera.com:27000), , , Instance 204c719f48777536:b83eb0ed0009 (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000a (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000b (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000c (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000d (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000e (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000f (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0010 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0011 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0012 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0013 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0014 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0015 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0016 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0017 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0018 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0019 (host=ia0322.halxg.cloudera.com:27000), , , , Instance 204c719f48777536:b83eb0ed001d (host=ia0322.halxg.cloudera.com:27000), , , , Instance 204c719f48777536:b83eb0ed0021 (host=ia0322.halxg.cloudera.com:27000), , , , Instance 204c719f48777536:b83eb0ed0025 (host=ia0317.halxg.cloudera.com:27000), , , , ...{code} Note several missing instances from host ia0306, ia0322, ia0317, and so on. I tried to tune FLAGS_status_report_interval_ms faster to every 500ms and there are even more instance profile missing. On the other hand, setting FLAGS_status_report_interval_ms=0 (only send final report) will result in complete runtime profile. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-10811) RPC to submit query getting stuck for AWS NLB forever.
[ https://issues.apache.org/jira/browse/IMPALA-10811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell reassigned IMPALA-10811: -- Assignee: Joe McDonnell > RPC to submit query getting stuck for AWS NLB forever. > -- > > Key: IMPALA-10811 > URL: https://issues.apache.org/jira/browse/IMPALA-10811 > Project: IMPALA > Issue Type: Bug >Reporter: Amogh Margoor >Assignee: Joe McDonnell >Priority: Major > Attachments: profile+(13).txt > > > Initial RPC to submit a query and fetch the query handle can take quite long > time to return as it can do various operations for planning and submission > that involve executing Catalog Operations like Rename, Alter Table Recover > partition that can take time on tables with many > partitions([https://github.com/apache/impala/blob/1231208da7104c832c13f272d1e5b8f554d29337/be/src/exec/catalog-op-executor.cc#L92]). > Attached is the profile of one such DDL query (with few fields hidden). > These RPCs are: > 1. Beeswax: > [https://github.com/apache/impala/blob/b28da054f3595bb92873433211438306fc22fbc7/be/src/service/impala-beeswax-server.cc#L57] > 2. HS2: > [https://github.com/apache/impala/blob/b28da054f3595bb92873433211438306fc22fbc7/be/src/service/impala-hs2-server.cc#L462] > > One of the side effects of such RPC taking long time is that clients such as > impala-shell using AWS NLB can get stuck for ever. The reason is NLB tracks > and closes connections after 350s and cannot be configured. But after closing > the connection it doesn;t send TCP RST to the client. Only when client tries > to send data or packets NLB issues back TCP RST to indicate connection is not > alive. Documentation is here: > [https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html#connection-idle-timeout]. > Hence the impala-shell waiting for RPC to return gets stuck indefinitely. > Hence, we may need to evaluate techniques for RPCs to return query handle > after > # Creating Driver: > [https://github.com/apache/impala/blob/b28da054f3595bb92873433211438306fc22fbc7/be/src/service/impala-server.cc#L1150] > # Register Query: > [https://github.com/apache/impala/blob/b28da054f3595bb92873433211438306fc22fbc7/be/src/service/impala-server.cc#L1168] > and execute later parts of RPC asynchronously in different thread without > blocking the RPC. That way clients can get query handle and poll for it for > state and results. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-10883) Dense runtime profile missing some fragment instance profile
[ https://issues.apache.org/jira/browse/IMPALA-10883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riza Suminto reassigned IMPALA-10883: - Assignee: Riza Suminto > Dense runtime profile missing some fragment instance profile > > > Key: IMPALA-10883 > URL: https://issues.apache.org/jira/browse/IMPALA-10883 > Project: IMPALA > Issue Type: Bug > Components: Distributed Exec >Affects Versions: Impala 4.0.0 >Reporter: Riza Suminto >Assignee: Riza Suminto >Priority: Major > > I ran TPC-DS Q78 in the following setup: > * Cluster of 20 nodes > * MT_DOP=12 > * --gen_experimental_profile=true > When I check the query profile of the completed query, I notice that couple > fragment are missing profile from some instances. This missing profile is > indicated in the missing instances id like this: > {code:java} > Fragment F00 [228 instances]: > Instances: Instance 204c719f48777536:b83eb0ed0001 > (host=ia0306.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed0002 (host=ia0306.halxg.cloudera.com:27000), , > Instance 204c719f48777536:b83eb0ed0004 > (host=ia0306.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed0005 (host=ia0306.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed0006 > (host=ia0306.halxg.cloudera.com:27000), , , Instance > 204c719f48777536:b83eb0ed0009 (host=ia0306.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed000a > (host=ia0306.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed000b (host=ia0306.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed000c > (host=ia0306.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed000d (host=ia0318.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed000e > (host=ia0318.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed000f (host=ia0318.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed0010 > (host=ia0318.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed0011 (host=ia0318.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed0012 > (host=ia0318.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed0013 (host=ia0318.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed0014 > (host=ia0318.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed0015 (host=ia0318.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed0016 > (host=ia0318.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed0017 (host=ia0318.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed0018 > (host=ia0318.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed0019 (host=ia0322.halxg.cloudera.com:27000), , , > , Instance 204c719f48777536:b83eb0ed001d > (host=ia0322.halxg.cloudera.com:27000), , , , Instance > 204c719f48777536:b83eb0ed0021 (host=ia0322.halxg.cloudera.com:27000), , , > , Instance 204c719f48777536:b83eb0ed0025 > (host=ia0317.halxg.cloudera.com:27000), , , , ...{code} > Note several missing instances from host ia0306, ia0322, ia0317, and so on. > I tried to tune FLAGS_status_report_interval_ms faster to every 500ms and > there are even more instance profile missing. On the other hand, setting > FLAGS_status_report_interval_ms=0 (only send final report) will result in > complete runtime profile. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10884) Improve pretty-printing of fragment instance name
Riza Suminto created IMPALA-10884: - Summary: Improve pretty-printing of fragment instance name Key: IMPALA-10884 URL: https://issues.apache.org/jira/browse/IMPALA-10884 Project: IMPALA Issue Type: Improvement Components: Backend Affects Versions: Impala 4.0.0 Reporter: Riza Suminto Assignee: Riza Suminto When using impala-profile-tool with extended verbosity, SummaryStatsCounter and TimeSeriesCounter coming from different instances are printed with prefix number like this: {code:java} - MemoryUsage[0] (2s000ms): 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 57.26 MB, 62.01 MB, 63.80 MB, 63.80 MB, 63.80 MB, 63.78 MB, 63.08 MB, 61.77 MB, 60.47 MB, 60.44 MB, 60.62 MB, 61.08 MB, 61.32 MB, 61.21 MB, 61.32 MB, 61.28 MB, 61.45 MB, 61.35 MB, 61.30 MB, 61.32 MB, 61.43 MB, 61.28 MB, 61.56 MB, 61.49 MB, 61.47 MB, 61.50 MB, 61.47 MB, 61.64 MB, 61.49 MB, 61.54 MB, 61.42 MB, 61.47 MB, 61.51 MB - MemoryUsage[1] (2s000ms): 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 32.64 MB, 57.42 MB, 63.74 MB, 63.74 MB, 63.74 MB, 63.74 MB, 63.89 MB, 63.41 MB, 61.98 MB, 62.72 MB, 61.88 MB, 61.24 MB, 61.09 MB, 61.15 MB, 61.17 MB, 61.47 MB, 61.63 MB, 61.68 MB, 61.38 MB, 61.57 MB, 61.58 MB, 61.44 MB, 61.61 MB, 61.62 MB, 61.37 MB, 61.64 MB, 61.63 MB, 61.36 MB, 61.70 MB, 61.49 MB, 61.45 MB, 61.48 MB, 61.50 MB, 61.44 MB - MemoryUsage[3] (2s000ms): 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 39.72 MB, 45.50 MB, 49.18 MB, 52.27 MB, 52.27 MB, 51.80 MB, 49.63 MB, 49.91 MB, 48.78 MB, 48.87 MB, 48.72 MB, 49.47 MB, 49.40 MB, 49.01 MB, 61.53 MB, 61.35 MB, 61.29 MB, 61.27 MB, 61.43 MB, 61.58 MB, 61.59 MB, 61.58 MB, 61.54 MB, 61.58 MB, 61.40 MB, 61.44 MB, 61.47 MB, 61.50 MB, 61.53 MB, 61.62 MB, 61.15 MB, 61.63 MB, 61.62 MB ... {code} The prefix number maps into fragment instance from where that counter is coming from. However, the fragment instance name is still printed in single long line without prefix number like this: Fragment F00 [228 instances]: Instances: Instance 204c719f48777536:b83eb0ed0001 (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0002 (host=ia0306.halxg.cloudera.com:27000), , Instance 204c719f48777536:b83eb0ed0004 (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0005 (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0006 (host=ia0306.halxg.cloudera.com:27000), , , Instance 204c719f48777536:b83eb0ed0009 (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000a (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000b (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000c (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000d (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000e (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000f (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0010 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0011 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0012 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0013 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0014 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0015 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0016 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0017 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0018 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0019 (host=ia0322.halxg.cloudera.com:27000), , , , Instance 204c719f48777536:b83eb0ed001d (host=ia0322.halxg.cloudera.com:27000), , , , Instance 204c719f48777536:b83eb0ed0021 (host=ia0322.halxg.cloudera.com:27000), , , , Instance 204c719f48777536:b83eb0ed0025 (host=ia0317.halxg.cloudera.com:27000), , , , ... This makes it hard to match counters with their fragment instance. We should break thing long line into multiple line and add prefix number. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-10884) Improve pretty-printing of fragment instance name
[ https://issues.apache.org/jira/browse/IMPALA-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riza Suminto updated IMPALA-10884: -- Description: When using impala-profile-tool with extended verbosity, SummaryStatsCounter and TimeSeriesCounter coming from different instances are printed with prefix number like this: {code:java} - MemoryUsage[0] (2s000ms): 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 57.26 MB, 62.01 MB, 63.80 MB, 63.80 MB, 63.80 MB, 63.78 MB, 63.08 MB, 61.77 MB, 60.47 MB, 60.44 MB, 60.62 MB, 61.08 MB, 61.32 MB, 61.21 MB, 61.32 MB, 61.28 MB, 61.45 MB, 61.35 MB, 61.30 MB, 61.32 MB, 61.43 MB, 61.28 MB, 61.56 MB, 61.49 MB, 61.47 MB, 61.50 MB, 61.47 MB, 61.64 MB, 61.49 MB, 61.54 MB, 61.42 MB, 61.47 MB, 61.51 MB - MemoryUsage[1] (2s000ms): 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 32.64 MB, 57.42 MB, 63.74 MB, 63.74 MB, 63.74 MB, 63.74 MB, 63.89 MB, 63.41 MB, 61.98 MB, 62.72 MB, 61.88 MB, 61.24 MB, 61.09 MB, 61.15 MB, 61.17 MB, 61.47 MB, 61.63 MB, 61.68 MB, 61.38 MB, 61.57 MB, 61.58 MB, 61.44 MB, 61.61 MB, 61.62 MB, 61.37 MB, 61.64 MB, 61.63 MB, 61.36 MB, 61.70 MB, 61.49 MB, 61.45 MB, 61.48 MB, 61.50 MB, 61.44 MB - MemoryUsage[3] (2s000ms): 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 39.72 MB, 45.50 MB, 49.18 MB, 52.27 MB, 52.27 MB, 51.80 MB, 49.63 MB, 49.91 MB, 48.78 MB, 48.87 MB, 48.72 MB, 49.47 MB, 49.40 MB, 49.01 MB, 61.53 MB, 61.35 MB, 61.29 MB, 61.27 MB, 61.43 MB, 61.58 MB, 61.59 MB, 61.58 MB, 61.54 MB, 61.58 MB, 61.40 MB, 61.44 MB, 61.47 MB, 61.50 MB, 61.53 MB, 61.62 MB, 61.15 MB, 61.63 MB, 61.62 MB ... {code} The prefix number maps into fragment instance from where that counter is coming from. However, the fragment instance name is still printed in single long line without prefix number like this: {code:java} Fragment F00 [228 instances]: Instances: Instance 204c719f48777536:b83eb0ed0001 (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0002 (host=ia0306.halxg.cloudera.com:27000), , Instance 204c719f48777536:b83eb0ed0004 (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0005 (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0006 (host=ia0306.halxg.cloudera.com:27000), , , Instance 204c719f48777536:b83eb0ed0009 (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000a (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000b (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000c (host=ia0306.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000d (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000e (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed000f (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0010 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0011 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0012 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0013 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0014 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0015 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0016 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0017 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0018 (host=ia0318.halxg.cloudera.com:27000), Instance 204c719f48777536:b83eb0ed0019 (host=ia0322.halxg.cloudera.com:27000), , , , Instance 204c719f48777536:b83eb0ed001d (host=ia0322.halxg.cloudera.com:27000), , , , Instance 204c719f48777536:b83eb0ed0021 (host=ia0322.halxg.cloudera.com:27000), , , , Instance 204c719f48777536:b83eb0ed0025 (host=ia0317.halxg.cloudera.com:27000), , , , ...{code} This makes it hard to match counters with their fragment instance. We should break thing long line into multiple line and add prefix number. was: When using impala-profile-tool with extended verbosity, SummaryStatsCounter and TimeSeriesCounter coming from different instances are printed with prefix number like this: {code:java} - MemoryUsage[0] (2s000ms): 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 57.26 MB, 62.01 MB, 63.80 MB, 63.80 MB, 63.80 MB, 63.78 MB, 63.08 MB, 61.77 MB, 60.47 MB, 60.44 MB, 60.62 MB, 61.08 MB, 61.32 MB, 61.21 MB, 61.32 MB, 61.28 MB, 61.45 MB, 61.35 MB, 61.30 MB, 61.32 MB, 61.43 MB, 61.28 MB, 61.56 MB, 61.49 MB, 61.47 MB, 61.50 MB, 61.47 MB, 61.64 MB, 61.49 MB, 61.54 MB, 61.42 MB, 61.47 MB, 61.51 MB - MemoryUsage[1] (2s000ms): 24.64 MB, 24.64 MB, 24.64 MB, 24.64 MB, 32.64 MB, 57.42 MB, 63.74 MB, 63.74 MB, 63.74 MB, 63.74 MB, 63.89 MB, 63.41 MB, 61.98 MB, 62.72 MB, 61.88 MB, 61.24
[jira] [Commented] (IMPALA-10883) Dense runtime profile missing some fragment instance profile
[ https://issues.apache.org/jira/browse/IMPALA-10883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17404982#comment-17404982 ] Riza Suminto commented on IMPALA-10883: --- There seem to be different expectations between coordinator and reporter backend. Let say MT_DOP=12 and a backend sending runtime profile update several times. Coordinator update its backend state profile through function call BackendState::ApplyExecStatusReport() -> AggregatedRuntimeProfile::UpdateAggregatedFromInstances(). UpdateAggregatedFromInstances() expects that the incoming TRuntimeProfileTree contains all 12 runtime profiles from 12 instances that run at the reporting backend. [https://github.com/apache/impala/blob/237ed5e/be/src/util/runtime-profile.cc#L542-L544] However, on the sender side, the backend can send a partial update. That is when some of the fragment instances finished earlier and already send their final status update. The agg_profile is initialized with 12 runtime profiles slots. But the slot that belongs to the finished fragment instance will not be populated. [https://github.com/apache/impala/blob/237ed5e/be/src/runtime/query-state.cc#L558-L575] > Dense runtime profile missing some fragment instance profile > > > Key: IMPALA-10883 > URL: https://issues.apache.org/jira/browse/IMPALA-10883 > Project: IMPALA > Issue Type: Bug > Components: Distributed Exec >Affects Versions: Impala 4.0.0 >Reporter: Riza Suminto >Assignee: Riza Suminto >Priority: Major > > I ran TPC-DS Q78 in the following setup: > * Cluster of 20 nodes > * MT_DOP=12 > * --gen_experimental_profile=true > When I check the query profile of the completed query, I notice that couple > fragment are missing profile from some instances. This missing profile is > indicated in the missing instances id like this: > {code:java} > Fragment F00 [228 instances]: > Instances: Instance 204c719f48777536:b83eb0ed0001 > (host=ia0306.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed0002 (host=ia0306.halxg.cloudera.com:27000), , > Instance 204c719f48777536:b83eb0ed0004 > (host=ia0306.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed0005 (host=ia0306.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed0006 > (host=ia0306.halxg.cloudera.com:27000), , , Instance > 204c719f48777536:b83eb0ed0009 (host=ia0306.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed000a > (host=ia0306.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed000b (host=ia0306.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed000c > (host=ia0306.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed000d (host=ia0318.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed000e > (host=ia0318.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed000f (host=ia0318.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed0010 > (host=ia0318.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed0011 (host=ia0318.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed0012 > (host=ia0318.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed0013 (host=ia0318.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed0014 > (host=ia0318.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed0015 (host=ia0318.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed0016 > (host=ia0318.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed0017 (host=ia0318.halxg.cloudera.com:27000), > Instance 204c719f48777536:b83eb0ed0018 > (host=ia0318.halxg.cloudera.com:27000), Instance > 204c719f48777536:b83eb0ed0019 (host=ia0322.halxg.cloudera.com:27000), , , > , Instance 204c719f48777536:b83eb0ed001d > (host=ia0322.halxg.cloudera.com:27000), , , , Instance > 204c719f48777536:b83eb0ed0021 (host=ia0322.halxg.cloudera.com:27000), , , > , Instance 204c719f48777536:b83eb0ed0025 > (host=ia0317.halxg.cloudera.com:27000), , , , ...{code} > Note several missing instances from host ia0306, ia0322, ia0317, and so on. > I tried to tune FLAGS_status_report_interval_ms faster to every 500ms and > there are even more instance profile missing. On the other hand, setting > FLAGS_status_report_interval_ms=0 (only send final report) will result in > complete runtime profile. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org