[GitHub] incubator-carbondata pull request #123: [CARBONDATA-204] Clear queryStatisti...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/123 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #123: [CARBONDATA-204] Clear queryStatisti...
Github user foryou2030 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/123#discussion_r77511283
--- Diff:
core/src/main/java/org/apache/carbondata/core/carbon/querystatistics/DriverQueryStatisticsRecorder.java
---
@@ -78,106 +83,148 @@ public synchronized void
recordStatisticsForDriver(QueryStatistic statistic, Str
*/
public void logStatisticsAsTableDriver() {
synchronized (lock) {
- String tableInfo = collectDriverStatistics();
- if (null != tableInfo) {
-LOGGER.statistic(tableInfo);
+ Iterator>> entries =
+ queryStatisticsMap.entrySet().iterator();
+ while (entries.hasNext()) {
+Map.Entry> entry = entries.next();
+String queryId = entry.getKey();
+// clear the unknown query statistics
+if(StringUtils.isEmpty(queryId)) {
+ queryStatisticsMap.remove(queryId);
--- End diff --
ok, handled
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #123: [CARBONDATA-204] Clear queryStatisti...
Github user foryou2030 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/123#discussion_r77506707
--- Diff:
core/src/main/java/org/apache/carbondata/core/carbon/querystatistics/DriverQueryStatisticsRecorder.java
---
@@ -78,106 +83,148 @@ public synchronized void
recordStatisticsForDriver(QueryStatistic statistic, Str
*/
public void logStatisticsAsTableDriver() {
synchronized (lock) {
- String tableInfo = collectDriverStatistics();
- if (null != tableInfo) {
-LOGGER.statistic(tableInfo);
+ Iterator>> entries =
+ queryStatisticsMap.entrySet().iterator();
+ while (entries.hasNext()) {
+Map.Entry> entry = entries.next();
+String queryId = entry.getKey();
+// clear the unknown query statistics
+if(StringUtils.isEmpty(queryId)) {
+ queryStatisticsMap.remove(queryId);
--- End diff --
I tried this, but caused some exceptions
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #123: [CARBONDATA-204] Clear queryStatisti...
Github user Vimal-Das commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/123#discussion_r77496731
--- Diff:
core/src/main/java/org/apache/carbondata/core/carbon/querystatistics/DriverQueryStatisticsRecorder.java
---
@@ -78,106 +83,148 @@ public synchronized void
recordStatisticsForDriver(QueryStatistic statistic, Str
*/
public void logStatisticsAsTableDriver() {
synchronized (lock) {
- String tableInfo = collectDriverStatistics();
- if (null != tableInfo) {
-LOGGER.statistic(tableInfo);
+ Iterator>> entries =
+ queryStatisticsMap.entrySet().iterator();
+ while (entries.hasNext()) {
+Map.Entry> entry = entries.next();
+String queryId = entry.getKey();
+// clear the unknown query statistics
+if(StringUtils.isEmpty(queryId)) {
+ queryStatisticsMap.remove(queryId);
--- End diff --
use the Iterator.remove() for better safety
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #123: [CARBONDATA-204] Clear queryStatisti...
Github user foryou2030 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/123#discussion_r77488586
--- Diff:
core/src/main/java/org/apache/carbondata/core/carbon/querystatistics/DriverQueryStatisticsRecorder.java
---
@@ -78,106 +82,142 @@ public synchronized void
recordStatisticsForDriver(QueryStatistic statistic, Str
*/
public void logStatisticsAsTableDriver() {
synchronized (lock) {
- String tableInfo = collectDriverStatistics();
- if (null != tableInfo) {
-LOGGER.statistic(tableInfo);
+ for (String key: queryStatisticsMap.keySet()) {
+// print
sql_parse_t,load_meta_t,block_allocation_t,block_identification_t
+// or just print block_allocation_t,block_identification_t
+if (queryStatisticsMap.get(key).size() >= 2) {
--- End diff --
ok, thanks
handled, pls check
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #123: [CARBONDATA-204] Clear queryStatisti...
Github user foryou2030 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/123#discussion_r77488614
--- Diff:
core/src/main/java/org/apache/carbondata/core/carbon/querystatistics/DriverQueryStatisticsRecorder.java
---
@@ -78,106 +82,142 @@ public synchronized void
recordStatisticsForDriver(QueryStatistic statistic, Str
*/
public void logStatisticsAsTableDriver() {
synchronized (lock) {
- String tableInfo = collectDriverStatistics();
- if (null != tableInfo) {
-LOGGER.statistic(tableInfo);
+ for (String key: queryStatisticsMap.keySet()) {
+// print
sql_parse_t,load_meta_t,block_allocation_t,block_identification_t
+// or just print block_allocation_t,block_identification_t
+if (queryStatisticsMap.get(key).size() >= 2) {
+ String tableInfo = collectDriverStatistics(key);
+ if (null != tableInfo) {
+LOGGER.statistic(tableInfo);
+ }
+}
+// clear timeout query statistics
+if(StringUtils.isEmpty(key)) {
+ queryStatisticsMap.remove(key);
+} else {
+ long interval = System.nanoTime() - Long.parseLong(key);
+ if (interval >
QueryStatisticsConstants.CLEAR_STATISTICS_TIMEOUT) {
+queryStatisticsMap.remove(key);
+ }
+}
}
}
}
/**
* Below method will parse queryStatisticsMap and put time into table
*/
- public String collectDriverStatistics() {
-for (String key: queryStatisticsMap.keySet()) {
- try {
-// TODO: get the finished query, and print Statistics
-if (queryStatisticsMap.get(key).size() > 3) {
- String sql_parse_time = "";
- String load_meta_time = "";
- String block_allocation_time = "";
- String block_identification_time = "";
- Double driver_part_time_tmp = 0.0;
- String splitChar = " ";
- // get statistic time from the QueryStatistic
- for (QueryStatistic statistic : queryStatisticsMap.get(key)) {
-switch (statistic.getMessage()) {
- case QueryStatisticsConstants.SQL_PARSE:
-sql_parse_time += statistic.getTimeTaken() + splitChar;
-driver_part_time_tmp += statistic.getTimeTaken();
-break;
- case QueryStatisticsConstants.LOAD_META:
-load_meta_time += statistic.getTimeTaken() + splitChar;
-driver_part_time_tmp += statistic.getTimeTaken();
-break;
- case QueryStatisticsConstants.BLOCK_ALLOCATION:
-block_allocation_time += statistic.getTimeTaken() +
splitChar;
-driver_part_time_tmp += statistic.getTimeTaken();
-break;
- case QueryStatisticsConstants.BLOCK_IDENTIFICATION:
-block_identification_time += statistic.getTimeTaken() +
splitChar;
-driver_part_time_tmp += statistic.getTimeTaken();
-break;
- default:
-break;
-}
- }
- String driver_part_time = driver_part_time_tmp + splitChar;
- // structure the query statistics info table
- StringBuilder tableInfo = new StringBuilder();
- int len1 = 8;
- int len2 = 20;
- int len3 = 21;
- int len4 = 22;
- String line = "+" + printLine("-", len1) + "+" + printLine("-",
len2) + "+" +
- printLine("-", len3) + "+" + printLine("-", len4) + "+";
- String line2 = "|" + printLine(" ", len1) + "+" + printLine("-",
len2) + "+" +
- printLine(" ", len3) + "+" + printLine("-", len4) + "+";
- // table header
- tableInfo.append(line).append("\n");
- tableInfo.append("|" + printLine(" ", (len1 -
"Module".length())) + "Module" + "|" +
- printLine(" ", (len2 - "Operation Step".length())) +
"Operation Step" + "|" +
- printLine(" ", (len3 + len4 + 1 - "Query Cost".length())) +
- "Query Cost" + "|" + "\n");
- // driver part
- tableInfo.append(line).append("\n");
- tableInfo.append("|" + printLine(" ", len1) + "|" +
- printLine(" ", (len2 - "SQL parse".length())) + "SQL parse"
+ "|" +
- printLine(" ", len3) + "|" +
- printLine(" ", (len4 - sql_parse_time.length())) +
sql_parse_time + "|" + "\n");
- tableInfo.append(line2).append("\n");
- tableInfo.append("|" +printLine(" ", (len1 - "Driver".length()))
+ "Driver" + "|"
[GitHub] incubator-carbondata pull request #123: [CARBONDATA-204] Clear queryStatisti...
Github user Vimal-Das commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/123#discussion_r77435578
--- Diff:
core/src/main/java/org/apache/carbondata/core/carbon/querystatistics/DriverQueryStatisticsRecorder.java
---
@@ -78,106 +82,142 @@ public synchronized void
recordStatisticsForDriver(QueryStatistic statistic, Str
*/
public void logStatisticsAsTableDriver() {
synchronized (lock) {
- String tableInfo = collectDriverStatistics();
- if (null != tableInfo) {
-LOGGER.statistic(tableInfo);
+ for (String key: queryStatisticsMap.keySet()) {
+// print
sql_parse_t,load_meta_t,block_allocation_t,block_identification_t
+// or just print block_allocation_t,block_identification_t
+if (queryStatisticsMap.get(key).size() >= 2) {
+ String tableInfo = collectDriverStatistics(key);
+ if (null != tableInfo) {
+LOGGER.statistic(tableInfo);
+ }
+}
+// clear timeout query statistics
+if(StringUtils.isEmpty(key)) {
+ queryStatisticsMap.remove(key);
+} else {
+ long interval = System.nanoTime() - Long.parseLong(key);
+ if (interval >
QueryStatisticsConstants.CLEAR_STATISTICS_TIMEOUT) {
+queryStatisticsMap.remove(key);
+ }
+}
}
}
}
/**
* Below method will parse queryStatisticsMap and put time into table
*/
- public String collectDriverStatistics() {
-for (String key: queryStatisticsMap.keySet()) {
- try {
-// TODO: get the finished query, and print Statistics
-if (queryStatisticsMap.get(key).size() > 3) {
- String sql_parse_time = "";
- String load_meta_time = "";
- String block_allocation_time = "";
- String block_identification_time = "";
- Double driver_part_time_tmp = 0.0;
- String splitChar = " ";
- // get statistic time from the QueryStatistic
- for (QueryStatistic statistic : queryStatisticsMap.get(key)) {
-switch (statistic.getMessage()) {
- case QueryStatisticsConstants.SQL_PARSE:
-sql_parse_time += statistic.getTimeTaken() + splitChar;
-driver_part_time_tmp += statistic.getTimeTaken();
-break;
- case QueryStatisticsConstants.LOAD_META:
-load_meta_time += statistic.getTimeTaken() + splitChar;
-driver_part_time_tmp += statistic.getTimeTaken();
-break;
- case QueryStatisticsConstants.BLOCK_ALLOCATION:
-block_allocation_time += statistic.getTimeTaken() +
splitChar;
-driver_part_time_tmp += statistic.getTimeTaken();
-break;
- case QueryStatisticsConstants.BLOCK_IDENTIFICATION:
-block_identification_time += statistic.getTimeTaken() +
splitChar;
-driver_part_time_tmp += statistic.getTimeTaken();
-break;
- default:
-break;
-}
- }
- String driver_part_time = driver_part_time_tmp + splitChar;
- // structure the query statistics info table
- StringBuilder tableInfo = new StringBuilder();
- int len1 = 8;
- int len2 = 20;
- int len3 = 21;
- int len4 = 22;
- String line = "+" + printLine("-", len1) + "+" + printLine("-",
len2) + "+" +
- printLine("-", len3) + "+" + printLine("-", len4) + "+";
- String line2 = "|" + printLine(" ", len1) + "+" + printLine("-",
len2) + "+" +
- printLine(" ", len3) + "+" + printLine("-", len4) + "+";
- // table header
- tableInfo.append(line).append("\n");
- tableInfo.append("|" + printLine(" ", (len1 -
"Module".length())) + "Module" + "|" +
- printLine(" ", (len2 - "Operation Step".length())) +
"Operation Step" + "|" +
- printLine(" ", (len3 + len4 + 1 - "Query Cost".length())) +
- "Query Cost" + "|" + "\n");
- // driver part
- tableInfo.append(line).append("\n");
- tableInfo.append("|" + printLine(" ", len1) + "|" +
- printLine(" ", (len2 - "SQL parse".length())) + "SQL parse"
+ "|" +
- printLine(" ", len3) + "|" +
- printLine(" ", (len4 - sql_parse_time.length())) +
sql_parse_time + "|" + "\n");
- tableInfo.append(line2).append("\n");
- tableInfo.append("|" +printLine(" ", (len1 - "Driver".length()))
+ "Driver" + "|" +
[GitHub] incubator-carbondata pull request #123: [CARBONDATA-204] Clear queryStatisti...
Github user Vimal-Das commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/123#discussion_r77435539
--- Diff:
core/src/main/java/org/apache/carbondata/core/carbon/querystatistics/DriverQueryStatisticsRecorder.java
---
@@ -78,106 +82,142 @@ public synchronized void
recordStatisticsForDriver(QueryStatistic statistic, Str
*/
public void logStatisticsAsTableDriver() {
synchronized (lock) {
- String tableInfo = collectDriverStatistics();
- if (null != tableInfo) {
-LOGGER.statistic(tableInfo);
+ for (String key: queryStatisticsMap.keySet()) {
+// print
sql_parse_t,load_meta_t,block_allocation_t,block_identification_t
+// or just print block_allocation_t,block_identification_t
+if (queryStatisticsMap.get(key).size() >= 2) {
--- End diff --
The call can return null, because the keyset obtained reflects the state of
map when the .keySet() method was called, later changes in the map will not be
reflected. So, if the element is removed in the meantime, get() can return null.
Solution: use an iterator over entrySet()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
