[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r379504306
##
File path:
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/DurationUtils.java
##
@@ -213,4 +213,36 @@ private DateTime parseToDateTime(Downsampling
downsampling, long time) {
}
throw new UnexpectedException("Unexpected downsampling: " +
downsampling.name());
}
+
+public DateTime startTimeBucket2DateTime(Downsampling downsampling, long
startTB) {
+switch (downsampling) {
+case Month:
+return MM.parseDateTime(String.valueOf(startTB));
+case Day:
+return MMDD.parseDateTime(String.valueOf(startTB));
+case Hour:
+return MMDDHH.parseDateTime(String.valueOf(startTB));
+case Minute:
+return MMDDHHMM.parseDateTime(String.valueOf(startTB));
+case Second:
+return MMDDHHMMSS.parseDateTime(String.valueOf(startTB));
+}
+throw new UnexpectedException("Unexpected downsampling: " +
downsampling.name());
+}
+
+public DateTime endTimeBucket2DateTime(Downsampling downsampling, long
endTB) {
+switch (downsampling) {
+case Month:
+return MM.parseDateTime(String.valueOf(endTB));
+case Day:
+return MMDD.parseDateTime(String.valueOf(endTB));
+case Hour:
+return MMDDHH.parseDateTime(String.valueOf(endTB));
+case Minute:
+return MMDDHHMM.parseDateTime(String.valueOf(endTB));
+case Second:
+return MMDDHHMMSS.parseDateTime(String.valueOf(endTB));
+}
+throw new UnexpectedException("Unexpected downsampling: " +
downsampling.name());
Review comment:
ack, for the day and month, different circulation methods are used.
Considering that the monthly increase calculation should be time-consuming, it
is still the original method. WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket URL: https://github.com/apache/skywalking/pull/4353#discussion_r379502634 ## File path: oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java ## @@ -291,6 +291,24 @@ public SearchResponse search(String indexName, SearchSourceBuilder searchSourceB return client.search(searchRequest); } +/** + * Search results from ES search engine according to various search conditions, + * Note the method is usered for the list of index names is optimized based on + * the scope of startTimeBucket and endTimeBucket + * @param indexNameList full index names list base on timebucket scope. + * Except for endpoint_inventory network_address_inventory service_inventory service_instance_inventory + * @param searchSourceBuilder Various search query conditions + * @return ES search query results + * @throws IOException throw IOException Review comment: ack This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r379381981
##
File path:
oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##
@@ -403,4 +421,11 @@ public String formatIndexName(String indexName) {
}
return indexName;
}
+
+public List formatIndexNames(List indexNameList) {
+if (StringUtil.isNotEmpty(namespace)) {
+indexNameList.stream().map(indexName -> namespace + "_" +
indexName);
Review comment:
oops, ack
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r379375965
##
File path:
oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/base/EsModelName.java
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base;
+
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.skywalking.oap.server.core.analysis.Downsampling;
+import org.apache.skywalking.oap.server.core.query.DurationUtils;
+import org.apache.skywalking.oap.server.core.register.EndpointInventory;
+import org.apache.skywalking.oap.server.core.register.NetworkAddressInventory;
+import org.apache.skywalking.oap.server.core.register.ServiceInstanceInventory;
+import org.apache.skywalking.oap.server.core.register.ServiceInventory;
+import org.apache.skywalking.oap.server.core.storage.model.ModelName;
+import org.joda.time.DateTime;
+import org.joda.time.format.DateTimeFormat;
+import org.joda.time.format.DateTimeFormatter;
+
+public class EsModelName extends ModelName {
+
+private static final DateTimeFormatter MM =
DateTimeFormat.forPattern("MM");
+private static final DateTimeFormatter MMDD =
DateTimeFormat.forPattern("MMdd");
+
+public static final List WHITE_INDEX_LIST = new
ArrayList() {
+{
+add(EndpointInventory.INDEX_NAME);
+add(NetworkAddressInventory.INDEX_NAME);
+add(ServiceInventory.INDEX_NAME);
+add(ServiceInstanceInventory.INDEX_NAME);
+}
+};
Review comment:
ack
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r379373679
##
File path:
oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/base/EsModelName.java
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base;
+
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.skywalking.oap.server.core.analysis.Downsampling;
+import org.apache.skywalking.oap.server.core.query.DurationUtils;
+import org.apache.skywalking.oap.server.core.register.EndpointInventory;
+import org.apache.skywalking.oap.server.core.register.NetworkAddressInventory;
+import org.apache.skywalking.oap.server.core.register.ServiceInstanceInventory;
+import org.apache.skywalking.oap.server.core.register.ServiceInventory;
+import org.apache.skywalking.oap.server.core.storage.model.ModelName;
+import org.joda.time.DateTime;
+import org.joda.time.format.DateTimeFormat;
+import org.joda.time.format.DateTimeFormatter;
+
+public class EsModelName extends ModelName {
+
+private static final DateTimeFormatter MM =
DateTimeFormat.forPattern("MM");
+private static final DateTimeFormatter MMDD =
DateTimeFormat.forPattern("MMdd");
+
+public static final List WHITE_INDEX_LIST = new
ArrayList() {
+{
+add(EndpointInventory.INDEX_NAME);
+add(NetworkAddressInventory.INDEX_NAME);
+add(ServiceInventory.INDEX_NAME);
+add(ServiceInstanceInventory.INDEX_NAME);
+}
+};
+
+public static List build(Downsampling downsampling, String
modelName, long startTB, long endTB) {
+if (WHITE_INDEX_LIST.contains(modelName)) {
+return new ArrayList() {
+{
+add(modelName);
+}
+};
Review comment:
ack
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r379368436
##
File path:
oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/base/EsModelName.java
##
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base;
+
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.skywalking.oap.server.core.analysis.Downsampling;
+import org.apache.skywalking.oap.server.core.query.DurationUtils;
+import org.apache.skywalking.oap.server.core.register.EndpointInventory;
+import org.apache.skywalking.oap.server.core.register.NetworkAddressInventory;
+import org.apache.skywalking.oap.server.core.register.ServiceInstanceInventory;
+import org.apache.skywalking.oap.server.core.register.ServiceInventory;
+import org.apache.skywalking.oap.server.core.storage.model.ModelName;
+import org.joda.time.DateTime;
+import org.joda.time.format.DateTimeFormat;
+import org.joda.time.format.DateTimeFormatter;
+
+public class EsModelName extends ModelName {
+
+private static final DateTimeFormatter MM =
DateTimeFormat.forPattern("MM");
+private static final DateTimeFormatter MMDD =
DateTimeFormat.forPattern("MMdd");
+
+public static final List WHITE_INDEX_LIST = new
ArrayList() {
+{
+add(EndpointInventory.INDEX_NAME);
+add(NetworkAddressInventory.INDEX_NAME);
+add(ServiceInventory.INDEX_NAME);
+add(ServiceInstanceInventory.INDEX_NAME);
+}
+};
+
+public static List build(Downsampling downsampling, String
modelName, long startTB, long endTB) {
+if (WHITE_INDEX_LIST.contains(modelName)) {
+return new ArrayList() {
+{
+add(modelName);
+}
+};
+}
+
+List indexNameList = new LinkedList<>();
+DateTime startDT =
DurationUtils.INSTANCE.startTimeBucket2DateTime(downsampling, startTB);
+DateTime endDT =
DurationUtils.INSTANCE.endTimeBucket2DateTime(downsampling, endTB);
+if (endDT.isAfter(startDT)) {
+switch (downsampling) {
+case Month:
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MM.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusMonths(1);
+}
+break;
+case Day:
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+case Hour:
+//current hour index is also suffix with MMDD
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+case Minute:
+//current minute index is also suffix with MMDD
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+case Second:
+//current second index is also suffix with MMDD
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378636611
##
File path:
oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##
@@ -291,6 +292,30 @@ public SearchResponse search(String indexName,
SearchSourceBuilder searchSourceB
return client.search(searchRequest);
}
+public SearchResponse search(String[] indexNames, SearchSourceBuilder
searchSourceBuilder) throws IOException {
+String[] fullIndexNames = formatIndexNames(indexNames);
+SearchRequest searchRequest = new SearchRequest(fullIndexNames);
+searchRequest.types(TYPE);
+searchRequest.source(searchSourceBuilder);
+return client.search(searchRequest);
+}
+
+public String[] filterNotExistIndex(String[] fullIndexNames, String
indName) throws IOException {
+// if no wrap, it is impossible to remove elements
+List indexNameList = new
ArrayList<>(Arrays.asList(fullIndexNames));
+if (fullIndexNames.length > 0) {
+List existIndex = retrievalIndexByAliases(indName);
+indexNameList.removeIf(indexName -> {
+//only filter index name with -
+if (indexName.contains("-")) {
Review comment:
server-core module depends on library-client module, so `Const#LINE` in
server-core can not used in library-client module.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378636611
##
File path:
oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##
@@ -291,6 +292,30 @@ public SearchResponse search(String indexName,
SearchSourceBuilder searchSourceB
return client.search(searchRequest);
}
+public SearchResponse search(String[] indexNames, SearchSourceBuilder
searchSourceBuilder) throws IOException {
+String[] fullIndexNames = formatIndexNames(indexNames);
+SearchRequest searchRequest = new SearchRequest(fullIndexNames);
+searchRequest.types(TYPE);
+searchRequest.source(searchSourceBuilder);
+return client.search(searchRequest);
+}
+
+public String[] filterNotExistIndex(String[] fullIndexNames, String
indName) throws IOException {
+// if no wrap, it is impossible to remove elements
+List indexNameList = new
ArrayList<>(Arrays.asList(fullIndexNames));
+if (fullIndexNames.length > 0) {
+List existIndex = retrievalIndexByAliases(indName);
+indexNameList.removeIf(indexName -> {
+//only filter index name with -
+if (indexName.contains("-")) {
Review comment:
server-core module depends on server-client module, so `Const#LINE` in
server-core can not used in server-client module.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378636611
##
File path:
oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##
@@ -291,6 +292,30 @@ public SearchResponse search(String indexName,
SearchSourceBuilder searchSourceB
return client.search(searchRequest);
}
+public SearchResponse search(String[] indexNames, SearchSourceBuilder
searchSourceBuilder) throws IOException {
+String[] fullIndexNames = formatIndexNames(indexNames);
+SearchRequest searchRequest = new SearchRequest(fullIndexNames);
+searchRequest.types(TYPE);
+searchRequest.source(searchSourceBuilder);
+return client.search(searchRequest);
+}
+
+public String[] filterNotExistIndex(String[] fullIndexNames, String
indName) throws IOException {
+// if no wrap, it is impossible to remove elements
+List indexNameList = new
ArrayList<>(Arrays.asList(fullIndexNames));
+if (fullIndexNames.length > 0) {
+List existIndex = retrievalIndexByAliases(indName);
+indexNameList.removeIf(indexName -> {
+//only filter index name with -
+if (indexName.contains("-")) {
Review comment:
server-core depends on server-client, so `Const#LINE` can not used in
server-client module.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378636611
##
File path:
oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##
@@ -291,6 +292,30 @@ public SearchResponse search(String indexName,
SearchSourceBuilder searchSourceB
return client.search(searchRequest);
}
+public SearchResponse search(String[] indexNames, SearchSourceBuilder
searchSourceBuilder) throws IOException {
+String[] fullIndexNames = formatIndexNames(indexNames);
+SearchRequest searchRequest = new SearchRequest(fullIndexNames);
+searchRequest.types(TYPE);
+searchRequest.source(searchSourceBuilder);
+return client.search(searchRequest);
+}
+
+public String[] filterNotExistIndex(String[] fullIndexNames, String
indName) throws IOException {
+// if no wrap, it is impossible to remove elements
+List indexNameList = new
ArrayList<>(Arrays.asList(fullIndexNames));
+if (fullIndexNames.length > 0) {
+List existIndex = retrievalIndexByAliases(indName);
+indexNameList.removeIf(indexName -> {
+//only filter index name with -
+if (indexName.contains("-")) {
Review comment:
server-core module depends on server-client module, so `Const#LINE` can not
used in server-client module.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378257166
##
File path:
oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##
@@ -291,6 +292,30 @@ public SearchResponse search(String indexName,
SearchSourceBuilder searchSourceB
return client.search(searchRequest);
}
+public SearchResponse search(String[] indexNames, SearchSourceBuilder
searchSourceBuilder) throws IOException {
+String[] fullIndexNames = formatIndexNames(indexNames);
+SearchRequest searchRequest = new SearchRequest(fullIndexNames);
+searchRequest.types(TYPE);
+searchRequest.source(searchSourceBuilder);
+return client.search(searchRequest);
+}
+
+public String[] filterNotExistIndex(String[] fullIndexNames, String
indName) throws IOException {
+// if no wrap, it is impossible to remove elements
+List indexNameList = new
ArrayList<>(Arrays.asList(fullIndexNames));
+if (fullIndexNames.length > 0) {
+List existIndex = retrievalIndexByAliases(indName);
+indexNameList.removeIf(indexName -> {
+//only filter index name with -
+if (indexName.contains("-")) {
Review comment:
ACK
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378255741
##
File path:
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/ModelName.java
##
@@ -30,10 +43,73 @@ public static String build(Downsampling downsampling,
String modelName) {
return modelName + Const.ID_SPLIT + Downsampling.Day.getName();
case Hour:
return modelName + Const.ID_SPLIT +
Downsampling.Hour.getName();
-//case Second:
-//return modelName + Const.ID_SPLIT +
Downsampling.Second.getName();
default:
return modelName;
}
}
+
+public static String[] build(Downsampling downsampling, String modelName,
long startTB, long endTB) {
+List indexNameList = new ArrayList<>();
+
+List whiteIndexList = new ArrayList() {
+{
+add(EndpointInventory.INDEX_NAME);
+add(NetworkAddressInventory.INDEX_NAME);
+add(ServiceInventory.INDEX_NAME);
+add(ServiceInstanceInventory.INDEX_NAME);
+}
+};
+
+if (whiteIndexList.contains(modelName)) {
+return new String[] { modelName };
+}
+
+DateTime startDT =
DurationUtils.INSTANCE.startTimeBucket2DateTime(downsampling, startTB);
+DateTime endDT =
DurationUtils.INSTANCE.endTimeBucket2DateTime(downsampling, endTB);
+if (endDT.isAfter(startDT)) {
+switch (downsampling) {
+case Month:
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MM.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusMonths(1);
+}
+break;
+case Day:
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+case Hour:
+//current hour index is also suffix with MMDD
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+case Minute:
+//current minute index is also suffix with MMDD
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+case Second:
+//current second index is also suffix with MMDD
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+default:
+indexNameList.add(modelName);
+}
+}
+return indexNameList.toArray(new String[0]);
Review comment:
> Did you answer the question in the wrong context? I am not saying
`filterNotExistIndex` is useless.
Oh, Let me perfect it. Convert at last minitue.
> But based on the things you talked, I think you should establish a cache
to `filterNotExistIndex` and cache the result for at least a day, considering
the index could only be deleted day by day, right? Set up a guava cache with
`#expireAfterWrite`?
Good advice, i will try.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378255741
##
File path:
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/ModelName.java
##
@@ -30,10 +43,73 @@ public static String build(Downsampling downsampling,
String modelName) {
return modelName + Const.ID_SPLIT + Downsampling.Day.getName();
case Hour:
return modelName + Const.ID_SPLIT +
Downsampling.Hour.getName();
-//case Second:
-//return modelName + Const.ID_SPLIT +
Downsampling.Second.getName();
default:
return modelName;
}
}
+
+public static String[] build(Downsampling downsampling, String modelName,
long startTB, long endTB) {
+List indexNameList = new ArrayList<>();
+
+List whiteIndexList = new ArrayList() {
+{
+add(EndpointInventory.INDEX_NAME);
+add(NetworkAddressInventory.INDEX_NAME);
+add(ServiceInventory.INDEX_NAME);
+add(ServiceInstanceInventory.INDEX_NAME);
+}
+};
+
+if (whiteIndexList.contains(modelName)) {
+return new String[] { modelName };
+}
+
+DateTime startDT =
DurationUtils.INSTANCE.startTimeBucket2DateTime(downsampling, startTB);
+DateTime endDT =
DurationUtils.INSTANCE.endTimeBucket2DateTime(downsampling, endTB);
+if (endDT.isAfter(startDT)) {
+switch (downsampling) {
+case Month:
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MM.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusMonths(1);
+}
+break;
+case Day:
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+case Hour:
+//current hour index is also suffix with MMDD
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+case Minute:
+//current minute index is also suffix with MMDD
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+case Second:
+//current second index is also suffix with MMDD
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+default:
+indexNameList.add(modelName);
+}
+}
+return indexNameList.toArray(new String[0]);
Review comment:
> You covert to array at the first place, but then you covert to List again
in `#filterNotExistIndex`. Seems unnecessary?
Oh, indeed i will fix it. Convert at last minitue.
> But based on the things you talked, I think you should establish a cache
to `filterNotExistIndex` and cache the result for at least a day, considering
the index could only be deleted day by day, right? Set up a guava cache with
`#expireAfterWrite`?
Good advice, i will try.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378255741
##
File path:
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/ModelName.java
##
@@ -30,10 +43,73 @@ public static String build(Downsampling downsampling,
String modelName) {
return modelName + Const.ID_SPLIT + Downsampling.Day.getName();
case Hour:
return modelName + Const.ID_SPLIT +
Downsampling.Hour.getName();
-//case Second:
-//return modelName + Const.ID_SPLIT +
Downsampling.Second.getName();
default:
return modelName;
}
}
+
+public static String[] build(Downsampling downsampling, String modelName,
long startTB, long endTB) {
+List indexNameList = new ArrayList<>();
+
+List whiteIndexList = new ArrayList() {
+{
+add(EndpointInventory.INDEX_NAME);
+add(NetworkAddressInventory.INDEX_NAME);
+add(ServiceInventory.INDEX_NAME);
+add(ServiceInstanceInventory.INDEX_NAME);
+}
+};
+
+if (whiteIndexList.contains(modelName)) {
+return new String[] { modelName };
+}
+
+DateTime startDT =
DurationUtils.INSTANCE.startTimeBucket2DateTime(downsampling, startTB);
+DateTime endDT =
DurationUtils.INSTANCE.endTimeBucket2DateTime(downsampling, endTB);
+if (endDT.isAfter(startDT)) {
+switch (downsampling) {
+case Month:
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MM.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusMonths(1);
+}
+break;
+case Day:
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+case Hour:
+//current hour index is also suffix with MMDD
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+case Minute:
+//current minute index is also suffix with MMDD
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+case Second:
+//current second index is also suffix with MMDD
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+default:
+indexNameList.add(modelName);
+}
+}
+return indexNameList.toArray(new String[0]);
Review comment:
> You covert to array at the first place, but then you covert to List again
in `#filterNotExistIndex`. Seems unnecessary?
Oh, indeed i will fix it. Convert at last minitue.
> But based on the things you talked, I think you should establish a cache
to `filterNotExistIndex` and cache the result for at least a day, considering
the index could only be deleted day by day, right? Set up a guava cache with
`#expireAfterWrite`?
Good advice, i will try.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378198260
##
File path:
oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##
@@ -291,6 +292,30 @@ public SearchResponse search(String indexName,
SearchSourceBuilder searchSourceB
return client.search(searchRequest);
}
+public SearchResponse search(String[] indexNames, SearchSourceBuilder
searchSourceBuilder) throws IOException {
Review comment:
good idea
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378197175
##
File path:
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/ModelName.java
##
@@ -30,10 +43,73 @@ public static String build(Downsampling downsampling,
String modelName) {
return modelName + Const.ID_SPLIT + Downsampling.Day.getName();
case Hour:
return modelName + Const.ID_SPLIT +
Downsampling.Hour.getName();
-//case Second:
-//return modelName + Const.ID_SPLIT +
Downsampling.Second.getName();
default:
return modelName;
}
}
+
+public static String[] build(Downsampling downsampling, String modelName,
long startTB, long endTB) {
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378197273
##
File path:
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/ModelName.java
##
@@ -30,10 +43,73 @@ public static String build(Downsampling downsampling,
String modelName) {
return modelName + Const.ID_SPLIT + Downsampling.Day.getName();
case Hour:
return modelName + Const.ID_SPLIT +
Downsampling.Hour.getName();
-//case Second:
-//return modelName + Const.ID_SPLIT +
Downsampling.Second.getName();
default:
return modelName;
}
}
+
+public static String[] build(Downsampling downsampling, String modelName,
long startTB, long endTB) {
+List indexNameList = new ArrayList<>();
+
+List whiteIndexList = new ArrayList() {
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378179210
##
File path:
oap-server/server-library/library-client/src/main/java/org/apache/skywalking/oap/server/library/client/elasticsearch/ElasticSearchClient.java
##
@@ -291,6 +292,30 @@ public SearchResponse search(String indexName,
SearchSourceBuilder searchSourceB
return client.search(searchRequest);
}
+public SearchResponse search(String[] indexNames, SearchSourceBuilder
searchSourceBuilder) throws IOException {
+String[] fullIndexNames = formatIndexNames(indexNames);
+SearchRequest searchRequest = new SearchRequest(fullIndexNames);
+searchRequest.types(TYPE);
+searchRequest.source(searchSourceBuilder);
+return client.search(searchRequest);
+}
+
+public String[] filterNotExistIndex(String[] fullIndexNames, String
indName) throws IOException {
+// if no wrap, it is impossible to remove elements
+List indexNameList = new
ArrayList<>(Arrays.asList(fullIndexNames));
+if (fullIndexNames.length > 0) {
+List existIndex = retrievalIndexByAliases(indName);
+indexNameList.removeIf(indexName -> {
+//only filter index name with -
+if (indexName.contains("-")) {
Review comment:
for example `segment alarm_record`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378153034
##
File path:
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/model/ModelName.java
##
@@ -30,10 +43,73 @@ public static String build(Downsampling downsampling,
String modelName) {
return modelName + Const.ID_SPLIT + Downsampling.Day.getName();
case Hour:
return modelName + Const.ID_SPLIT +
Downsampling.Hour.getName();
-//case Second:
-//return modelName + Const.ID_SPLIT +
Downsampling.Second.getName();
default:
return modelName;
}
}
+
+public static String[] build(Downsampling downsampling, String modelName,
long startTB, long endTB) {
+List indexNameList = new ArrayList<>();
+
+List whiteIndexList = new ArrayList() {
+{
+add(EndpointInventory.INDEX_NAME);
+add(NetworkAddressInventory.INDEX_NAME);
+add(ServiceInventory.INDEX_NAME);
+add(ServiceInstanceInventory.INDEX_NAME);
+}
+};
+
+if (whiteIndexList.contains(modelName)) {
+return new String[] { modelName };
+}
+
+DateTime startDT =
DurationUtils.INSTANCE.startTimeBucket2DateTime(downsampling, startTB);
+DateTime endDT =
DurationUtils.INSTANCE.endTimeBucket2DateTime(downsampling, endTB);
+if (endDT.isAfter(startDT)) {
+switch (downsampling) {
+case Month:
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MM.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusMonths(1);
+}
+break;
+case Day:
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+case Hour:
+//current hour index is also suffix with MMDD
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+case Minute:
+//current minute index is also suffix with MMDD
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+case Second:
+//current second index is also suffix with MMDD
+while (endDT.isAfter(startDT)) {
+String indexName = build(downsampling, modelName) +
"-" + MMDD.print(startDT);
+indexNameList.add(indexName);
+startDT = startDT.plusDays(1);
+}
+break;
+default:
+indexNameList.add(modelName);
+}
+}
+return indexNameList.toArray(new String[0]);
Review comment:
If select a date with a long span, which dates are not in the middle, or the
system has been stopped for a few days, there may be no index data for several
days. If you query date contain those days directly, searching the index will
appear `index not found exception`. so, Did a filter that the date index does
not exist
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378089483
##
File path:
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/DurationUtils.java
##
@@ -213,4 +213,36 @@ private DateTime parseToDateTime(Downsampling
downsampling, long time) {
}
throw new UnexpectedException("Unexpected downsampling: " +
downsampling.name());
}
+
+public DateTime startTimeBucket2DateTime(Downsampling downsampling, long
startTB) {
Review comment:
When crossing months, you need to consider the big and small months. In
addition, you need to consider some leap years. like `20200227-20200305`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378083903
##
File path:
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/DurationUtils.java
##
@@ -213,4 +213,36 @@ private DateTime parseToDateTime(Downsampling
downsampling, long time) {
}
throw new UnexpectedException("Unexpected downsampling: " +
downsampling.name());
}
+
+public DateTime startTimeBucket2DateTime(Downsampling downsampling, long
startTB) {
Review comment:
Indeed, it maybe faster to convert to long like 202002. When the timebucket
scope in loop statement add one day or month will trouble. Or maybe convert to
timestamps，comparing and adding will fast，but also has to convert to Date
string.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378083903
##
File path:
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/DurationUtils.java
##
@@ -213,4 +213,36 @@ private DateTime parseToDateTime(Downsampling
downsampling, long time) {
}
throw new UnexpectedException("Unexpected downsampling: " +
downsampling.name());
}
+
+public DateTime startTimeBucket2DateTime(Downsampling downsampling, long
startTB) {
Review comment:
Indeed, it maybe faster to convert to long like 202002. When the timebucket
scope in loop statement add one day or month will trouble.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [skywalking] aderm commented on a change in pull request #4353: Optimizing performance reduces es index queries scope by timebucket
aderm commented on a change in pull request #4353: Optimizing performance
reduces es index queries scope by timebucket
URL: https://github.com/apache/skywalking/pull/4353#discussion_r378083903
##
File path:
oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/query/DurationUtils.java
##
@@ -213,4 +213,36 @@ private DateTime parseToDateTime(Downsampling
downsampling, long time) {
}
throw new UnexpectedException("Unexpected downsampling: " +
downsampling.name());
}
+
+public DateTime startTimeBucket2DateTime(Downsampling downsampling, long
startTB) {
Review comment:
Indeed, it maybe faster to convert to long. When the date while loop
statement is added, it needs to be converted to the character format of the
date.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
