[GitHub] [skywalking] aderm commented on issue #4292: Improve ES query performance
aderm commented on issue #4292: Improve ES query performance URL: https://github.com/apache/skywalking/pull/4292#issuecomment-584141118 Reference #4291, so closed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [skywalking] aderm commented on issue #4292: Improve ES query performance
aderm commented on issue #4292: Improve ES query performance URL: https://github.com/apache/skywalking/pull/4292#issuecomment-582887343 @kezhenxu94 Thanks for the feedback, something has happened at home these days and it was interrupted. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [skywalking] aderm commented on issue #4292: Improve ES query performance
aderm commented on issue #4292: Improve ES query performance URL: https://github.com/apache/skywalking/pull/4292#issuecomment-581047765 It is indeed a good idea to reduce the query index range, but in theory, there is no need to calculate the score, and there should be good improvements, including reducing the use of CPU resources This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [skywalking] aderm commented on issue #4292: Improve ES query performance
aderm commented on issue #4292: Improve ES query performance
URL: https://github.com/apache/skywalking/pull/4292#issuecomment-581047508
index doc count:612002
test code:
```
@Warmup(iterations = 5, time = 5, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 5, time = 5, timeUnit = TimeUnit.SECONDS)
@State(Scope.Benchmark)
public class FilterQueryTest {
private RestHighLevelClient client;
private final long startTimestamp = 1580287478790L;
private final long endTimestamp = 168048064L;
private final long MULTIPLE = 10L;
private final int SEARCH_SIZE = 5000;
private final String INDEX_NAME = "service_inventory";
@Setup
public void createClient() {
client = new RestHighLevelClient((RestClient.builder(new
HttpHost("127.0.0.1", 9200, "http";
}
@TearDown
public void closeClient() throws IOException {
client.close();
}
/**
* Benchmark test fix es query clause
*/
@Benchmark
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
public void testFixEsQuery() throws IOException {
SearchSourceBuilder sourceBuilder =
SearchSourceBuilder.searchSource();
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
boolQueryBuilder.must().add(timeRangeQueryBuild(startTimestamp,
endTimestamp));
boolQueryBuilder.must().add(QueryBuilders.termQuery("is_address",
0));
boolQueryBuilder.must().add(QueryBuilders.termQuery("node_type", 0));
sourceBuilder.query(boolQueryBuilder);
sourceBuilder.size(SEARCH_SIZE);
SearchRequest searchRequest = new SearchRequest(INDEX_NAME);
searchRequest.source(sourceBuilder);
SearchResponse response = client.search(searchRequest,
RequestOptions.DEFAULT);
}
/**
* Benchmark test fix es const query clause
*/
@Benchmark
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
public void testFixEsConstQuery() throws IOException {
SearchSourceBuilder sourceBuilder =
SearchSourceBuilder.searchSource();
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
boolQueryBuilder.must().add(timeRangeQueryBuild(startTimestamp,
endTimestamp));
boolQueryBuilder.must().add(QueryBuilders.termQuery("is_address",
0));
boolQueryBuilder.must().add(QueryBuilders.termQuery("node_type", 0));
sourceBuilder.query(new ConstantScoreQueryBuilder(boolQueryBuilder));
sourceBuilder.size(SEARCH_SIZE);
SearchRequest searchRequest = new SearchRequest(INDEX_NAME);
searchRequest.source(sourceBuilder);
SearchResponse response = client.search(searchRequest,
RequestOptions.DEFAULT);
}
/**
* Benchmark test fix es filter query clause
*/
@Benchmark
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
public void testFixEsFilterQuery() throws IOException {
SearchSourceBuilder sourceBuilder =
SearchSourceBuilder.searchSource();
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
boolQueryBuilder.must().add(timeRangeQueryBuild(startTimestamp,
endTimestamp));
boolQueryBuilder.must().add(QueryBuilders.termQuery("is_address",
0));
boolQueryBuilder.must().add(QueryBuilders.termQuery("node_type", 0));
sourceBuilder.query(QueryBuilders.boolQuery().filter(boolQueryBuilder));
sourceBuilder.size(SEARCH_SIZE);
SearchRequest searchRequest = new SearchRequest(INDEX_NAME);
searchRequest.source(sourceBuilder);
SearchResponse response = client.search(searchRequest,
RequestOptions.DEFAULT);
}
/**
* Benchmark test dynamic es query clause
*/
@Benchmark
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
public void testDynamicEsQuery() throws IOException {
SearchSourceBuilder sourceBuilder =
SearchSourceBuilder.searchSource();
long ranFactor = (long) (Math.random() * MULTIPLE);
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
boolQueryBuilder.must().add(timeRangeQueryBuild(startTimestamp +
ranFactor, endTimestamp + ranFactor));
boolQueryBuilder.must().add(QueryBuilders.termQuery("is_address",
0));
boolQueryBuilder.must().add(QueryBuilders.termQuery("node_type", 0));
sourceBuilder.query(boolQueryBuilder);
sourceBuilder.size(SEARCH_SIZE);
SearchRequest searchRequest = new SearchRequest(INDEX_NAME);
searchRequest.source(sourceBuilder);
Sear
[GitHub] [skywalking] aderm commented on issue #4292: Improve ES query performance
aderm commented on issue #4292: Improve ES query performance URL: https://github.com/apache/skywalking/pull/4292#issuecomment-581046491 Agree, Some additional data is added locally, but the test results change with the query, and the results are not fixed. Some additional data is added locally, but the test results change with the query, and the results are not fixed. The ConstantScoreQueryBuilder results are not ideal.There should be improvement in theory This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [skywalking] aderm commented on issue #4292: Improve ES query performance
aderm commented on issue #4292: Improve ES query performance URL: https://github.com/apache/skywalking/pull/4292#issuecomment-581041002 > @aderm My point is, this optimization seems not very obvious. Like you said, you are confused about the data, and if this really cached in the ES server, this would make more concerns. The time range keeps changes in different query, which could use more resources. OK, i will try another large amount of data. Regarding the cache occupying memory resources, really need to pay attention to it, look at how to release it, and the impact on performance. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [skywalking] aderm commented on issue #4292: Improve ES query performance
aderm commented on issue #4292: Improve ES query performance URL: https://github.com/apache/skywalking/pull/4292#issuecomment-581039237 Regarding the occasional failure of e2e, I checked the logs and found that the traces printed in org.apache.skywalking.e2e.SampleVerificationITCase #doRetryableVerification sometimes query size two, the expected value size is one.This will cause the test to fail. If the number of traces obtained by the query is 1, it will pass. Several times before the code has not changed, the overall e2e test passed again https://user-images.githubusercontent.com/2892433/73594327-61438c00-4548-11ea-91f7-206835ed7568.png";> @kezhenxu94 When executing concurrently, maybe this happen? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [skywalking] aderm commented on issue #4292: Improve ES query performance
aderm commented on issue #4292: Improve ES query performance URL: https://github.com/apache/skywalking/pull/4292#issuecomment-581033456 > Why `TraceQueryEsDAO` is not changed? I think the trace segment index has much more data than others. This is because there is a sort operation. If there is a custom sort, the query will not calculate the score. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [skywalking] aderm commented on issue #4292: Improve ES query performance
aderm commented on issue #4292: Improve ES query performance URL: https://github.com/apache/skywalking/pull/4292#issuecomment-579832206 timed out， A little strange. https://user-images.githubusercontent.com/2892433/73374322-f51d1a00-42f4-11ea-81e8-5926a69fe450.png";> This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
