[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16225936#comment-16225936 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-trafodion/pull/1279 > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > Fix For: 2.2-incubating > > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222625#comment-16222625 ] ASF GitHub Bot commented on TRAFODION-2420: --- GitHub user selvaganesang opened a pull request: https://github.com/apache/incubator-trafodion/pull/1279 [TRAFODION-2420] RMS Enhancements Refactored the ESP memory management so that all the SQL memory both heap and space objects are accounted correctly in RMS With this change, the total SQL memory used by all processes involved in the query is represented in the counter "SQL Heap WM" You can merge this pull request into a Git repository by running: $ git pull https://github.com/selvaganesang/incubator-trafodion trafodion-2420_9 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-trafodion/pull/1279.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1279 commit 32aaad0cbc7a3ba7cd590cfb689a94573750805b Author: selvaganesang Date: 2017-10-27T15:41:26Z [TRAFODION-2420] RMS Enhancements Refactored the ESP memory management so that all the SQL memory both heap and space objects are accounted correctly in RMS With this change, the total SQL memory used by all processes involved in the query is represented in the counter "SQL Heap WM" > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: New Feature > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > Fix For: 2.2-incubating > > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091921#comment-16091921 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-trafodion/pull/1184 > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089323#comment-16089323 ] ASF GitHub Bot commented on TRAFODION-2420: --- GitHub user selvaganesang opened a pull request: https://github.com/apache/incubator-trafodion/pull/1184 [TRAFODION-2420] RMS enhancements Added four new counters to provide information about multiple executes of a prepared statement. No. of times executed Min. Execute Time Max. Execute Time Avg. Execute Time You can merge this pull request into a Git repository by running: $ git pull https://github.com/selvaganesang/incubator-trafodion trafodion-2420_8 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-trafodion/pull/1184.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1184 commit 419c843aeb71b9912112067251b5574f1d6b05da Author: selvaganesang Date: 2017-07-17T05:13:32Z [TRAFODION-2420] RMS enhancements Added four new counters to provide information about multiple executes of a prepared statement. No. of times executed Min. Execute Time Max. Execute Time Avg. Execute Time > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16021513#comment-16021513 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-trafodion/pull/1101 > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020478#comment-16020478 ] ASF GitHub Bot commented on TRAFODION-2420: --- GitHub user selvaganesang opened a pull request: https://github.com/apache/incubator-trafodion/pull/1101 [TRAFODION-2420] RMS enhancements Fixed a possible corruption issue in Type 2 JDBC driver while getting the accumulated statistics. The SE_IO_Bytes is now changed to SE_IO_KBytes. It used to return in MB and now it is changed to KB. You can merge this pull request into a Git repository by running: $ git pull https://github.com/selvaganesang/incubator-trafodion trafodion-2420_5 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-trafodion/pull/1101.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1101 commit 7362068efe598a5dc27b4a55e77f15817926cd11 Author: selvaganesang Date: 2017-05-23T00:29:42Z [TRAFODION-2420] RMS enhancements Fixed a possible corruption issue in Type 2 JDBC driver while getting the accumulated statistics. The SE_IO_Bytes is now changed to SE_IO_KBytes. It used to return in MB and now it is changed to KB. > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005590#comment-16005590 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-trafodion/pull/1087 > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003661#comment-16003661
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1087#discussion_r115616713
--- Diff: core/sql/executor/ex_hash_grby.cpp ---
@@ -452,6 +463,8 @@ short ex_hash_grby_tcb::work() {
hasFreeTupp_ = TRUE;
};
};
+ if (bmoStats_)
+ bmoStats_->setBmoPhase(PHASE_END-HGB_READ_PHASE);
--- End diff --
Thanks for the explanation!
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003631#comment-16003631 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user selvaganesang commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/1087#discussion_r115612077 --- Diff: core/sql/executor/ex_hash_grby.h --- @@ -302,6 +312,7 @@ NA_EIDPROC void resetClusterAndReadFromChild(); // Tmobile. public: + static const char *HashGrbyPhaseStr[]; --- End diff -- That's correct. Static global variables are also thread safe and I am hoping these phases are not changed forever and hence it shouldn't be error prone. > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003624#comment-16003624
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1087#discussion_r115611428
--- Diff: core/sql/executor/ex_hash_grby.cpp ---
@@ -452,6 +463,8 @@ short ex_hash_grby_tcb::work() {
hasFreeTupp_ = TRUE;
};
};
+ if (bmoStats_)
+ bmoStats_->setBmoPhase(PHASE_END-HGB_READ_PHASE);
--- End diff --
With the lack of test case to test out the phase changes, I took a safer
approach of setting it many times. I will move this code to the two places
mentioned after I did some testing with TPCDS queries
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003611#comment-16003611
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1087#discussion_r115610081
--- Diff: core/sql/executor/ex_hash_grby.cpp ---
@@ -452,6 +463,8 @@ short ex_hash_grby_tcb::work() {
hasFreeTupp_ = TRUE;
};
};
+ if (bmoStats_)
+ bmoStats_->setBmoPhase(PHASE_END-HGB_READ_PHASE);
--- End diff --
For your latter question, I didn't want to disturb the existing enum values
and hence kept the existing value. But, for BMO stats, I wanted show the phase
of the instance that hasn't moved to the next higher phase. When the BMO stats
is merged from different esp instances, higher phase value is used. Hence, the
phase enum value is reversed for BMO stats
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003547#comment-16003547
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user zellerh commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1087#discussion_r115595056
--- Diff: core/sql/executor/ex_hash_grby.cpp ---
@@ -452,6 +463,8 @@ short ex_hash_grby_tcb::work() {
hasFreeTupp_ = TRUE;
};
};
+ if (bmoStats_)
+ bmoStats_->setBmoPhase(PHASE_END-HGB_READ_PHASE);
--- End diff --
This is executed for every row. Would it be cheaper to do this in the two
places where we set state_ to HASH_GRBY_READ_CHILD?
Another question not about correctness but more related to personal taste:
Why not simply assign values 1, 2, 3 etc. in the enum instead of subtracting
two enums here? Feel free to ignore if you like this solution better.
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003548#comment-16003548 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user zellerh commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/1087#discussion_r115593353 --- Diff: core/sql/executor/ex_hash_grby.h --- @@ -302,6 +312,7 @@ NA_EIDPROC void resetClusterAndReadFromChild(); // Tmobile. public: + static const char *HashGrbyPhaseStr[]; --- End diff -- Minor comment: One way to avoid the static, global variable would be to use a method that returns a constant rather than this array. > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001996#comment-16001996 ] ASF GitHub Bot commented on TRAFODION-2420: --- GitHub user selvaganesang opened a pull request: https://github.com/apache/incubator-trafodion/pull/1087 [TRAFODION-2420] RMS enhancements New counters introduced as part of BMO InterimRowCount - Counter to keep track intermediate rows produced by hash join and for intermediate runs in sort phase - BMO Phase for hash-join and sort get statistics for statement s1, options 'cs' now works. Removed attaching BMO stats to Non-root Paritial and leaf partition hash group by operators. These operators are non-blocking BMOs. Introduced 3 phases for hash grby operator to be visible in RMS Stats options 'sl' now works with get statistics for qid. The relevant new counters are now part of accumulated statistics type. Extended the support upto 512 opeartors from 256 operators in RMS Fixed both Type 2 and mxosrvr code to obtain all the externalized counters in accumualted stats correctly. You can merge this pull request into a Git repository by running: $ git pull https://github.com/selvaganesang/incubator-trafodion trafodion-2420_4 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-trafodion/pull/1087.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1087 commit e7719015286a0217fa5ec81551a783f8bc18fc3b Author: selvaganesang Date: 2017-05-01T23:11:47Z [TRAFODION-2420] RMS enhancements New counters introduced as part of BMO InterimRowCount - Counter to keep track intermediate rows produced by hash join and for intermediate runs in sort phase - BMO Phase for hash-join and sort get statistics for statement s1, options 'cs' now works. Removed attaching BMO stats to Non-root Paritial and leaf partition hash group by operators. These operators are non-blocking BMOs. Introduced 3 phases for hash grby operator to be visible in RMS Stats options 'sl' now works with get statistics for qid. The relevant new counters are now part of accumulated statistics type. Extended the support upto 512 opeartors from 256 operators in RMS Fixed both Type 2 and mxosrvr code to obtain all the externalized counters in accumualted stats correctly. > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15964878#comment-15964878 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-trafodion/pull/1052 > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15961294#comment-15961294 ] ASF GitHub Bot commented on TRAFODION-2420: --- GitHub user selvaganesang opened a pull request: https://github.com/apache/incubator-trafodion/pull/1052 [TRAFODION-2420] RMS enhancements Improved BMO operation accounting. Two new counters are added a) scratchIOSize - Size in KB of the scratch IO b) scratchIOMaxTime - Time in microseconds taken by any long pole ESP instance to do scratch IO operations The ScrBufSize, ScrBufRead and ScrBufWritten are no longer displayed in the formatted get statistics command outputs. Reduced the default value of CQD HBASE_NUM_CACHE_ROWS_MAX to 1024 from 1 to reduce stress on the client JVM memory. Disabled the memory pressure triggering in the BMO operators by increasing the threshold GEN_MEM_PRESSURE_THRESHOLD to 1 from 100. The memory pressure detection code was inadvertently enabled when the cap to limit memory pressure constant to 100 was removed. Memory pressure triggers in BMO operators will be enabled later when we understand the memory pressure detection better. You can merge this pull request into a Git repository by running: $ git pull https://github.com/selvaganesang/incubator-trafodion trafodion-2420_1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-trafodion/pull/1052.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1052 commit 0c10290402c121eec54541d3720c3f407155722a Author: selvaganesang Date: 2017-04-07T00:03:58Z [TRAFODION-2420] RMS enhancements Improved BMO operation accounting. Two new counters are added a) scratchIOSize - Size in KB of the scratch IO b) scratchIOMaxTime - Time in microseconds taken by any long pole ESP instance to do scratch IO operations The ScrBufSize, ScrBufRead and ScrBufWritten are no longer displayed in the formatted get statistics command outputs. Reduced the default value of CQD HBASE_NUM_CACHE_ROWS_MAX to 1024 from 1 to reduce stress on the client JVM memory. Disabled the memory pressure triggering in the BMO operators by increasing the threshold GEN_MEM_PRESSURE_THRESHOLD to 1 from 100. The memory pressure detection code was inadvertently enabled when the cap to limit memory pressure constant to 100 was removed. Memory pressure triggers in BMO operators will be enabled later when we understand the memory pressure detection better. > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945960#comment-15945960 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-trafodion/pull/1027 > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945693#comment-15945693
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1027#discussion_r108502285
--- Diff: core/sql/cli/sqlcli.h ---
@@ -1169,15 +1170,14 @@ enum SQLSTATS_ITEM_ID {
SQLSTATS_BMO_HEAP_WM = 152,
SQLSTATS_BMO_SPACE_BUFFER_SIZE = 153,
SQLSTATS_BMO_SPACE_BUFFER_COUNT = 154,
- SQLSTATS_OVEFLOW_PHASE_STARTTIME = 155,
- SQLSTATS_OVERFLOW_PHASE = 156,
SQLSTATS_SCRATCH_FILE_COUNT = 157,
SQLSTATS_SCRATCH_BUFFER_BLOCK_SIZE = 158,
SQLSTATS_SCRATCH_BUFFER_BLOCKS_READ = 159,
SQLSTATS_SCRATCH_BUFFER_BLOCKS_WRITTEN = 160,
SQLSTATS_SCRATCH_OVERFLOW_MODE = 161,
SQLSTATS_SCRATCH_READ_COUNT = 162,
SQLSTATS_SCRATCH_WRITE_COUNT = 163,
+ SQLSTATS_SCRATCH_IO_TIME = 155,
--- End diff --
I will move this entry to the deleted line below 1172 in a later check-in
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945675#comment-15945675
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/1027#discussion_r108498224
--- Diff: core/sql/cli/sqlcli.h ---
@@ -1169,15 +1170,14 @@ enum SQLSTATS_ITEM_ID {
SQLSTATS_BMO_HEAP_WM = 152,
SQLSTATS_BMO_SPACE_BUFFER_SIZE = 153,
SQLSTATS_BMO_SPACE_BUFFER_COUNT = 154,
- SQLSTATS_OVEFLOW_PHASE_STARTTIME = 155,
- SQLSTATS_OVERFLOW_PHASE = 156,
SQLSTATS_SCRATCH_FILE_COUNT = 157,
SQLSTATS_SCRATCH_BUFFER_BLOCK_SIZE = 158,
SQLSTATS_SCRATCH_BUFFER_BLOCKS_READ = 159,
SQLSTATS_SCRATCH_BUFFER_BLOCKS_WRITTEN = 160,
SQLSTATS_SCRATCH_OVERFLOW_MODE = 161,
SQLSTATS_SCRATCH_READ_COUNT = 162,
SQLSTATS_SCRATCH_WRITE_COUNT = 163,
+ SQLSTATS_SCRATCH_IO_TIME = 155,
--- End diff --
Might be better to keep these in numeric order? I'm wondering what might
happen if someone else added a new counter using 155, for example.
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944233#comment-15944233 ] ASF GitHub Bot commented on TRAFODION-2420: --- GitHub user selvaganesang opened a pull request: https://github.com/apache/incubator-trafodion/pull/1027 [TRAFODION-2420] RMS enhancements Introduced 2 new counters HeapWM: - "SQL Heap WM" - Tracks the high water mark of the used memory including space buffer and bmo operator memory. It is enough to monitor this counter alone to track the memory utilization of the query. scrIOTime: "Scr. IO Time" - Tracks the IO time taken to read/write to the scratch files at BMO operator level Introduced a 'Single line Format' option to display BMO stats and SE stats in a single line format to enable exporting the output to Excel easily. GET STATISTICS FOR QID CURRENT PROGRESS, OPTIONS 'SL' Removed and cleaned up the code further Also fixed memory corruption issues seen with GET STATISTICS command. You can merge this pull request into a Git repository by running: $ git pull https://github.com/selvaganesang/incubator-trafodion trafoddion-2420 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-trafodion/pull/1027.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1027 commit ae3eca8a46779828281143db01c171f052947d33 Author: selvaganesang Date: 2017-03-27T23:13:10Z [TRAFODION-2420] RMS enhancements Introduced 2 new counters HeapWM: - "SQL Heap WM" - Tracks the high water mark of the used memory including space buffer and bmo operator memory. It is enough to monitor this counter alone to track the memory utilization of the query. scrIOTime: "Scr. IO Time" - Tracks the IO time taken to read/write to the scratch files at BMO operator level Introduced a 'Single line Format' option to display BMO stats and SE stats in a single line format to enable exporting the output to Excel easily. GET STATISTICS FOR QID CURRENT PROGRESS, OPTIONS 'SL' Removed and cleaned up the code further Also fixed memory corruption issues seen with GET STATISTICS command. > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15834772#comment-15834772
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/918#discussion_r97344477
--- Diff: core/sql/executor/ExStats.h ---
@@ -3791,9 +3791,9 @@ NA_EIDPROC
short &cmpPriority() {return cmpPriority_;}
short &dp2Priority() {return dp2Priority_;}
short &fixupPriority() {return fixupPriority_;}
- inline void setNumSqlProcs(short i) {numSqlProcs_ = i; }
+ void incNumEspsInUse() { numOfTotalEspsUsed_++; }
inline void setNumCpus(short i) {numCpus_ = i; }
- inline short getNumSqlProcs() { return numSqlProcs_; }
+ inline short getNumSqlProcs() { return numOfTotalEspsUsed_+1; }
--- End diff --
Yes. Because if the query had just master, then the number of SQL procs
will be zero. Only the numOfTotalEspsUsed_ is maintained.
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15834765#comment-15834765
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/918#discussion_r97343889
--- Diff: core/sql/executor/ExStats.cpp ---
@@ -2741,13 +2749,21 @@ Int64 ExHbaseAccessStats::getNumVal(Int32 i) const
NABoolean ExHbaseAccessStats::filterForSEstats(struct timespec
currTimespec, Lng32 filter)
{
- blockTime_ = timer_.filterForSEstats(currTimespec);
- if (blockTime_ >= filter)
- return TRUE;
+ Int64 sumIOTime;
--- End diff --
Will take care of your suggestion, when this is done
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15834764#comment-15834764
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/918#discussion_r97343740
--- Diff: core/sql/executor/ExStats.cpp ---
@@ -2741,13 +2749,21 @@ Int64 ExHbaseAccessStats::getNumVal(Int32 i) const
NABoolean ExHbaseAccessStats::filterForSEstats(struct timespec
currTimespec, Lng32 filter)
{
- blockTime_ = timer_.filterForSEstats(currTimespec);
- if (blockTime_ >= filter)
- return TRUE;
+ Int64 sumIOTime;
--- End diff --
Yes. Can be done. I am thinking of getting rid of one of these classes
because both have same member variables sometime in future.
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15834638#comment-15834638
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/918#discussion_r97323080
--- Diff: core/sql/executor/ExStats.cpp ---
@@ -2357,11 +2357,19 @@ Int64 ExHdfsScanStats::getNumVal(Int32 i) const
NABoolean ExHdfsScanStats::filterForSEstats(struct timespec currTimespec,
Lng32 filter)
{
- blockTime_ = timer_.filterForSEstats(currTimespec);
- if (blockTime_ >= filter)
- return TRUE;
+ Int64 sumIOTime;
+
+ if (filter > 0) {
+ blockTime_ = timer_.filterForSEstats(currTimespec);
+ if (blockTime_ >= filter)
+ return TRUE;
+ }
else
- return FALSE;
+ if (queryId_ != NULL && (sumIOTime = timer_.getTime()) > 0 &&
(sumIOTime = sumIOTime /(100LL)) >= -filter) {
--- End diff --
Could write as sumIOTime /= 100LL
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15834639#comment-15834639
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/918#discussion_r97323697
--- Diff: core/sql/executor/ExStats.cpp ---
@@ -2741,13 +2749,21 @@ Int64 ExHbaseAccessStats::getNumVal(Int32 i) const
NABoolean ExHbaseAccessStats::filterForSEstats(struct timespec
currTimespec, Lng32 filter)
{
- blockTime_ = timer_.filterForSEstats(currTimespec);
- if (blockTime_ >= filter)
- return TRUE;
+ Int64 sumIOTime;
--- End diff --
The same logic seems duplicated between this class and ExHdfsScanStats.
Maybe some refactoring?
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15834637#comment-15834637 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user nonstop-qfchen commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/918#discussion_r97324599 --- Diff: core/sql/executor/ExStats.cpp --- @@ -2357,11 +2357,19 @@ Int64 ExHdfsScanStats::getNumVal(Int32 i) const NABoolean ExHdfsScanStats::filterForSEstats(struct timespec currTimespec, Lng32 filter) --- End diff -- I wonder if the type of filter argument should be Int64 here, since Int64 / 100LL could be a value larger than max(Int32). > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15834640#comment-15834640
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user nonstop-qfchen commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/918#discussion_r97324910
--- Diff: core/sql/executor/ExStats.h ---
@@ -3791,9 +3791,9 @@ NA_EIDPROC
short &cmpPriority() {return cmpPriority_;}
short &dp2Priority() {return dp2Priority_;}
short &fixupPriority() {return fixupPriority_;}
- inline void setNumSqlProcs(short i) {numSqlProcs_ = i; }
+ void incNumEspsInUse() { numOfTotalEspsUsed_++; }
inline void setNumCpus(short i) {numCpus_ = i; }
- inline short getNumSqlProcs() { return numSqlProcs_; }
+ inline short getNumSqlProcs() { return numOfTotalEspsUsed_+1; }
--- End diff --
Should we +1 every time we call getNumSqlProcs()?
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15829158#comment-15829158 ] ASF GitHub Bot commented on TRAFODION-2420: --- GitHub user selvaganesang opened a pull request: https://github.com/apache/incubator-trafodion/pull/918 [TRAFODION-2420] RMS Enhancements Added yet another offender feature to list query ids that has a total IO time for any storage engine opertor consuming longer than a given number of seconds. ./offender -s se_offender Will list the query ids along with the table name. SEE $TRAF_HOME/export/limited-support-tools/LSO/README The "Number of SQL Processes" counter is now made multi-fragment aware and hence contains the actual number of ESPs used + 1 for master process. You can merge this pull request into a Git repository by running: $ git pull https://github.com/selvaganesang/incubator-trafodion se_offender_1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-trafodion/pull/918.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #918 commit 45ff9540eeeacb46e4e65e2b6a57b0133760 Author: selvaganesang Date: 2017-01-19T01:58:56Z [TRAFODION-2420] RMS Enhancements Added yet another offender feature to list query ids that has a total IO time for any storage engine opertor consuming longer than a given number of seconds. ./offender -s se_offender Will list the query ids along with the table name. SEE $TRAF_HOME/export/limited-support-tools/LSO/README The "Number of SQL Processes" counter is now made multi-fragment aware and hence contains the actual number of ESPs used + 1 for master process. > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15804748#comment-15804748 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user selvaganesang closed the pull request at: https://github.com/apache/incubator-trafodion/pull/904 > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802957#comment-15802957 ] ASF GitHub Bot commented on TRAFODION-2420: --- GitHub user selvaganesang opened a pull request: https://github.com/apache/incubator-trafodion/pull/904 [TRAFODION-2420] RMS enhancements Fixed the issue pointed by Steve You can merge this pull request into a Git repository by running: $ git pull https://github.com/selvaganesang/incubator-trafodion se_offender Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-trafodion/pull/904.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #904 commit 461cdc879b887de437adee253fbeb74eef3204e3 Author: selvaganesang Date: 2017-01-06T00:02:31Z [TRAFODION-2420] RMS enhancements Fixed the issue pointed by Steve > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802941#comment-15802941
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94878400
--- Diff: core/sql/executor/ExExeUtilGetStats.cpp ---
@@ -2319,47 +2321,38 @@ short ExExeUtilGetRTSStatisticsTcb::work()
{
if (measStatsItems_ == NULL)
{
- maxMeasStatsItems_ = 35;
+ maxMeasStatsItems_ = 26;
--- End diff --
It looks like I looked at the wrong file ResStatisticsStatement.cpp when I
responded to Arvind. Hence I didn't catch the issue with duplicate index 22
measStatsItems_[22].statsItem_id = SQLSTATS_SCRATCH_BUFFER_BLOCKS_READ;
measStatsItems_[22].statsItem_id = SQLSTATS_SCRATCH_BUFFER_BLOCKS_WRITTEN;
measStatsItems_[23].statsItem_id = SQLSTATS_SCRATCH_READ_COUNT;
measStatsItems_[24].statsItem_id = SQLSTATS_SCRATCH_WRITE_COUNT;
I will fix it
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802772#comment-15802772
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user svarnau commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94868189
--- Diff: core/sql/executor/ExExeUtilGetStats.cpp ---
@@ -2319,47 +2321,38 @@ short ExExeUtilGetRTSStatisticsTcb::work()
{
if (measStatsItems_ == NULL)
{
- maxMeasStatsItems_ = 35;
+ maxMeasStatsItems_ = 26;
--- End diff --
Yes, that was the one!
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802754#comment-15802754
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94867234
--- Diff: core/sql/executor/ExExeUtilGetStats.cpp ---
@@ -2319,47 +2321,38 @@ short ExExeUtilGetRTSStatisticsTcb::work()
{
if (measStatsItems_ == NULL)
{
- maxMeasStatsItems_ = 35;
+ maxMeasStatsItems_ = 26;
--- End diff --
Where did you see this regression? Is it in core/TESTRTS?
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802647#comment-15802647
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user svarnau commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94860650
--- Diff: core/sql/executor/ExExeUtilGetStats.cpp ---
@@ -2319,47 +2321,38 @@ short ExExeUtilGetRTSStatisticsTcb::work()
{
if (measStatsItems_ == NULL)
{
- maxMeasStatsItems_ = 35;
+ maxMeasStatsItems_ = 26;
--- End diff --
I think Arvind was correct. The number should be 25. For me this caused a
regression failure by outputting a couple spurious stats at the end. (an extra
Accessed Rows stat where none should be)
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15800187#comment-15800187 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-trafodion/pull/898 > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798917#comment-15798917 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user selvaganesang commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/898#discussion_r94633228 --- Diff: core/sql/executor/ExStats.cpp --- @@ -3678,9 +3292,8 @@ void ExPartitionAccessStats::getVariableStatsInfo(char * dataBuffer, "AnsiName: %s PhysName: %s BuffersSize: %u BuffersSent: %u BuffersRcvd: %u NumMessages: %Ld MsgBytes: %Ld StatsBytes: %Ld MsgBytesSent: %Ld MsgBytesRcvd: %Ld ", ansiName_, fileName_, bufferStats()->sendBufferSize(), bufferStats()->sentBuffers().entryCnt(), bufferStats()->recdBuffers().entryCnt(), - fsDp2MsgsStats()->getNumMessages(), - fsDp2MsgsStats()->getMessageBytes(), - fsDp2MsgsStats()->getStatsBytes(), + exeSEStats()->getNumIOCalls(), + exeSEStats()->getNumIOBytes(), --- End diff -- Thanks Arvind. I have fixed the issue with ExPartitionAccess::getVariableStatsInfo to provide 0 for statsBytes. ExPartitionAccess is not used currently. Hence, the regression tests didn't detect this issue. Thanks for catching it. I have left ExPartitionAccess behind because it might be resurrected later > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798867#comment-15798867 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user arvind-narain commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/898#discussion_r94629541 --- Diff: core/sql/executor/ExStats.cpp --- @@ -3678,9 +3292,8 @@ void ExPartitionAccessStats::getVariableStatsInfo(char * dataBuffer, "AnsiName: %s PhysName: %s BuffersSize: %u BuffersSent: %u BuffersRcvd: %u NumMessages: %Ld MsgBytes: %Ld StatsBytes: %Ld MsgBytesSent: %Ld MsgBytesRcvd: %Ld ", ansiName_, fileName_, bufferStats()->sendBufferSize(), bufferStats()->sentBuffers().entryCnt(), bufferStats()->recdBuffers().entryCnt(), - fsDp2MsgsStats()->getNumMessages(), - fsDp2MsgsStats()->getMessageBytes(), - fsDp2MsgsStats()->getStatsBytes(), + exeSEStats()->getNumIOCalls(), + exeSEStats()->getNumIOBytes(), --- End diff -- Thanks Selva. Noticed that we have 10 format specifiers (including StatsBytes) and only 9 arguments hence the question. Line 4522 has similar logic ( though it's sprintf). Maybe I'm missing something here - will check on the code again as well as str_sprintf. > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798221#comment-15798221 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user selvaganesang commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/898#discussion_r94579003 --- Diff: core/sql/executor/ExStats.cpp --- @@ -3678,9 +3292,8 @@ void ExPartitionAccessStats::getVariableStatsInfo(char * dataBuffer, "AnsiName: %s PhysName: %s BuffersSize: %u BuffersSent: %u BuffersRcvd: %u NumMessages: %Ld MsgBytes: %Ld StatsBytes: %Ld MsgBytesSent: %Ld MsgBytesRcvd: %Ld ", ansiName_, fileName_, bufferStats()->sendBufferSize(), bufferStats()->sentBuffers().entryCnt(), bufferStats()->recdBuffers().entryCnt(), - fsDp2MsgsStats()->getNumMessages(), - fsDp2MsgsStats()->getMessageBytes(), - fsDp2MsgsStats()->getStatsBytes(), + exeSEStats()->getNumIOCalls(), + exeSEStats()->getNumIOBytes(), --- End diff -- Yes. It is supposed to denote how many stats bytes are transferred from SE to the executor periodically. But with the new SE, this concept is no longer applicable. > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798214#comment-15798214 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user selvaganesang commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/898#discussion_r94578723 --- Diff: core/conn/odbc/src/odbc/nsksrvrcore/ResStatisticsStatement.cpp --- @@ -1265,7 +1265,7 @@ void ResStatisticsStatement::setStatistics(SRVR_STMT_HDL *pSrvrStmt, SQLSTATS_TY #define MAX_PERTABLE_STATS_DESC30 #define MAX_MASTERSTATS_ENTRY 31 -#define MAX_MEASSTATS_ENTRY34 +#define MAX_MEASSTATS_ENTRY26 #define MAX_PERTABLE_ENTRY 14 --- End diff -- Yes. Changed it. Missed out in ODBC code but changed it in Type2 JDBC driver. I think there is a need to share this code between all components SQL, JDBC and ODBC. > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798215#comment-15798215
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94578756
--- Diff: core/sql/executor/ExExeUtilGetStats.cpp ---
@@ -2319,47 +2321,38 @@ short ExExeUtilGetRTSStatisticsTcb::work()
{
if (measStatsItems_ == NULL)
{
- maxMeasStatsItems_ = 35;
+ maxMeasStatsItems_ = 26;
--- End diff --
No, it should be 26
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796412#comment-15796412 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user arvind-narain commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/898#discussion_r94497176 --- Diff: core/sql/executor/ExStats.cpp --- @@ -3678,9 +3292,8 @@ void ExPartitionAccessStats::getVariableStatsInfo(char * dataBuffer, "AnsiName: %s PhysName: %s BuffersSize: %u BuffersSent: %u BuffersRcvd: %u NumMessages: %Ld MsgBytes: %Ld StatsBytes: %Ld MsgBytesSent: %Ld MsgBytesRcvd: %Ld ", ansiName_, fileName_, bufferStats()->sendBufferSize(), bufferStats()->sentBuffers().entryCnt(), bufferStats()->recdBuffers().entryCnt(), - fsDp2MsgsStats()->getNumMessages(), - fsDp2MsgsStats()->getMessageBytes(), - fsDp2MsgsStats()->getStatsBytes(), + exeSEStats()->getNumIOCalls(), + exeSEStats()->getNumIOBytes(), --- End diff -- 0 for StatsBytes ? > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[
https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796414#comment-15796414
]
ASF GitHub Bot commented on TRAFODION-2420:
---
Github user arvind-narain commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94496467
--- Diff: core/sql/executor/ExExeUtilGetStats.cpp ---
@@ -2319,47 +2321,38 @@ short ExExeUtilGetRTSStatisticsTcb::work()
{
if (measStatsItems_ == NULL)
{
- maxMeasStatsItems_ = 35;
+ maxMeasStatsItems_ = 26;
--- End diff --
Should this be 25 or 26 ?
> RMS enhancements
>
>
> Key: TRAFODION-2420
> URL: https://issues.apache.org/jira/browse/TRAFODION-2420
> Project: Apache Trafodion
> Issue Type: Improvement
> Components: sql-general
>Affects Versions: 2.1-incubating
>Reporter: Selvaganesan Govindarajan
>Assignee: Selvaganesan Govindarajan
>
> Improve RMS to provide the following features/capabilities:
> 1. Currently RMS can list queries that consumed cpu time between two time
> points. However, the calls to storage engine like hbase, hdfs etc are
> blocking and these blocked APIs will not allow the Trafodion SQL engine
> scheduler to detect the cpu time spent during the time the call was blocked.
> Introduce a feature called SE offender (storage engine offender) to list the
> queries that are blocked in storage engine more than a certain duration.
> 2. Remove counters that are no longer updated or needed
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15796413#comment-15796413 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user arvind-narain commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/898#discussion_r94495473 --- Diff: core/conn/odbc/src/odbc/nsksrvrcore/ResStatisticsStatement.cpp --- @@ -1265,7 +1265,7 @@ void ResStatisticsStatement::setStatistics(SRVR_STMT_HDL *pSrvrStmt, SQLSTATS_TY #define MAX_PERTABLE_STATS_DESC30 #define MAX_MASTERSTATS_ENTRY 31 -#define MAX_MEASSTATS_ENTRY34 +#define MAX_MEASSTATS_ENTRY26 #define MAX_PERTABLE_ENTRY 14 --- End diff -- Should MAX_PERTABLE_ENTRY here be 10 ? > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795636#comment-15795636 ] ASF GitHub Bot commented on TRAFODION-2420: --- Github user DaveBirdsall commented on a diff in the pull request: https://github.com/apache/incubator-trafodion/pull/898#discussion_r94448449 --- Diff: core/conn/jdbc_type2/native/ResStatisticsStatement.cpp --- @@ -1537,39 +1537,31 @@ void ResStatisticsStatement::setStatistics(SRVR_STMT_HDL *pSrvrStmt, SQLSTATS_TY initSqlStatsItems(measStatsItems_, MAX_MEASSTATS_ENTRY, &sqlStatsDesc_[currStatsDescEntry_]); measStatsItems_[0].statsItem_id = SQLSTATS_ACT_ROWS_ACCESSED; measStatsItems_[1].statsItem_id = SQLSTATS_ACT_ROWS_USED; -measStatsItems_[2].statsItem_id = SQLSTATS_MSG_COUNT; -measStatsItems_[3].statsItem_id = SQLSTATS_MSG_BYTES; -measStatsItems_[4].statsItem_id = SQLSTATS_STATS_BYTES; -measStatsItems_[5].statsItem_id = SQLSTATS_DISK_IOS; -measStatsItems_[6].statsItem_id = SQLSTATS_LOCK_WAITS; -measStatsItems_[7].statsItem_id = SQLSTATS_LOCK_ESCALATIONS; -measStatsItems_[8].statsItem_id = SQLSTATS_DP2_CPU_BUSY_TIME; -measStatsItems_[9].statsItem_id = SQLSTATS_SQL_CPU_BUSY_TIME; -measStatsItems_[10].statsItem_id = SQLSTATS_SQL_SPACE_ALLOC; -measStatsItems_[11].statsItem_id = SQLSTATS_SQL_SPACE_USED; -measStatsItems_[12].statsItem_id = SQLSTATS_SQL_HEAP_ALLOC; -measStatsItems_[13].statsItem_id = SQLSTATS_SQL_HEAP_USED; -measStatsItems_[14].statsItem_id = SQLSTATS_EID_SPACE_ALLOC; -measStatsItems_[15].statsItem_id = SQLSTATS_EID_SPACE_USED; -measStatsItems_[16].statsItem_id = SQLSTATS_EID_HEAP_ALLOC; -measStatsItems_[17].statsItem_id = SQLSTATS_EID_HEAP_USED; -measStatsItems_[18].statsItem_id = SQLSTATS_OPENS; -measStatsItems_[19].statsItem_id = SQLSTATS_OPEN_TIME; -measStatsItems_[20].statsItem_id = SQLSTATS_PROCESS_CREATED; -measStatsItems_[21].statsItem_id = SQLSTATS_PROCESS_CREATE_TIME; -measStatsItems_[22].statsItem_id = SQLSTATS_REQ_MSG_CNT; -measStatsItems_[23].statsItem_id = SQLSTATS_REQ_MSG_BYTES; -measStatsItems_[24].statsItem_id = SQLSTATS_REPLY_MSG_CNT; -measStatsItems_[25].statsItem_id = SQLSTATS_REPLY_MSG_BYTES; -measStatsItems_[26].statsItem_id = SQLSTATS_SCRATCH_FILE_COUNT; -measStatsItems_[27].statsItem_id = SQLSTATS_SCRATCH_BUFFER_BLOCK_SIZE; -measStatsItems_[28].statsItem_id = SQLSTATS_SCRATCH_BUFFER_BLOCKS_READ; -measStatsItems_[29].statsItem_id = SQLSTATS_SCRATCH_BUFFER_BLOCKS_WRITTEN; -measStatsItems_[30].statsItem_id = SQLSTATS_SCRATCH_OVERFLOW_MODE; -measStatsItems_[31].statsItem_id = SQLSTATS_SCRATCH_READ_COUNT; -measStatsItems_[32].statsItem_id = SQLSTATS_SCRATCH_WRITE_COUNT; -measStatsItems_[33].statsItem_id = SQLSTATS_UDR_CPU_BUSY_TIME; -// MAX_MEASSTATS_ENTRY is set to 34 +measStatsItems_[2].statsItem_id = SQLSTATS_SE_IOS; --- End diff -- Looks like you took all the old DP2-specific counters out and added some "SE" counters. Is my understanding correct? > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TRAFODION-2420) RMS enhancements
[ https://issues.apache.org/jira/browse/TRAFODION-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15783067#comment-15783067 ] ASF GitHub Bot commented on TRAFODION-2420: --- GitHub user selvaganesang opened a pull request: https://github.com/apache/incubator-trafodion/pull/898 [TRAFODION-2420] RMS enhancements Introduced a feature in RMS offender to list queries blocked in storage engine. This lists queries which are blocked in the storage engine APIs for more than given number of seconds. The BLOCKED_FOR_SECS denotes how long the storage engine API is blocked. The SQL commands to list SE_BLOCKED queries are available at: $TRAF_HOME/export/limited-support-tools/LSO/se_blocked.sql Example output is below: cd $TRAF_HOME/export/limited-support-tools/LSO ./offender -s se_blocked CURRENT_TIMESTAMP NO_OF_PROCESSES BLOCKED_FOR_SECS QUERY_ID TABLE_NAME -- --- -- 2016-12-28 10:29:43.941455183 MXID11305142123496807995800020206U300_18_S1 TRAFODION.SCH.T022 You can merge this pull request into a Git repository by running: $ git pull https://github.com/selvaganesang/incubator-trafodion se_offender Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-trafodion/pull/898.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #898 commit ce6b44612f3fc19a18314f985853322dd39b66e4 Author: selvaganesang Date: 2016-12-28T14:57:49Z [TRAFODION-2420] RMS enhancements Introduced a feature to list queries blocked in storage engine via RMS offender. This lists queries which are blocked in the storage engine APIs for more than given number of seconds. The BLOCKED_FOR_SECS denotes how long the storage engine API is blocked. The SQL commands to list SE_BLOCKED queries are available at: $TRAF_HOME/export/limited-support-tools/LSO/se_blocked.sql Example output is below: cd $TRAF_HOME/export/limited-support-tools/LSO ./offender -s se_blocked CURRENT_TIMESTAMP NO_OF_PROCESSES BLOCKED_FOR_SECS QUERY_ID TABLE_NAME -- --- -- 2016-12-28 10:29:43.941455183 MXID11305142123496807995800020206U300_18_S1 TRAFODION.SCH.T022 > RMS enhancements > > > Key: TRAFODION-2420 > URL: https://issues.apache.org/jira/browse/TRAFODION-2420 > Project: Apache Trafodion > Issue Type: Improvement > Components: sql-general >Affects Versions: 2.1-incubating >Reporter: Selvaganesan Govindarajan >Assignee: Selvaganesan Govindarajan > > Improve RMS to provide the following features/capabilities: > 1. Currently RMS can list queries that consumed cpu time between two time > points. However, the calls to storage engine like hbase, hdfs etc are > blocking and these blocked APIs will not allow the Trafodion SQL engine > scheduler to detect the cpu time spent during the time the call was blocked. > Introduce a feature called SE offender (storage engine offender) to list the > queries that are blocked in storage engine more than a certain duration. > 2. Remove counters that are no longer updated or needed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
