This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 070f42c463 [Enhancement](Es): Support config like whether push down to
es (#16800)
070f42c463 is described below
commit 070f42c4639caa0d62053f085009686e50f4a146
Author: Stalary <[email protected]>
AuthorDate: Fri Feb 17 21:56:11 2023 +0800
[Enhancement](Es): Support config like whether push down to es (#16800)
Support config like whether push down to es and refactor some code
Like transform to wildcard query and push down to es, this increases the
cpu consumption of the es,
I add a switch control it.
---
docs/en/docs/lakehouse/multi-catalog/es.md | 19 +-
docs/zh-CN/docs/lakehouse/multi-catalog/es.md | 19 +-
.../main/java/org/apache/doris/catalog/Env.java | 3 +-
.../java/org/apache/doris/catalog/EsResource.java | 7 +
.../java/org/apache/doris/catalog/EsTable.java | 12 +
.../doris/catalog/external/EsExternalTable.java | 1 +
.../apache/doris/datasource/EsExternalCatalog.java | 5 +
.../doris/external/elasticsearch/EsUtil.java | 203 ---------------
.../external/elasticsearch/QueryBuilders.java | 280 ++++++++++++++++++++-
.../java/org/apache/doris/planner/EsScanNode.java | 5 +-
.../doris/external/elasticsearch/EsUtilTest.java | 89 +++----
11 files changed, 367 insertions(+), 276 deletions(-)
diff --git a/docs/en/docs/lakehouse/multi-catalog/es.md
b/docs/en/docs/lakehouse/multi-catalog/es.md
index 0ad296899b..f44dc9cb62 100644
--- a/docs/en/docs/lakehouse/multi-catalog/es.md
+++ b/docs/en/docs/lakehouse/multi-catalog/es.md
@@ -50,16 +50,17 @@ After switching to the ES Catalog, you will be in the
`dafault_db` so you don't
### Parameter Description
-| Parameter | Required or Not | Default Value | Description
|
-| ----------------- | --------------- | ------------- |
------------------------------------------------------------ |
-| `hosts` | Yes | | ES address, can be one
or multiple addresses, or the load balancer address of ES |
-| `user` | No | Empty | ES username
|
-| `password` | No | Empty | Password of the
corresponding user |
-| `doc_value_scan` | No | true | Whether to obtain
value of the target field by ES/Lucene columnar storage |
-| `keyword_sniff` | No | true | Whether to sniff the
text.fields in ES based on keyword; If this is set to false, the system will
perform matching after tokenization. |
+| Parameter | Required or Not | Default Value | Description
|
+| ----------------- | --------------- | -------------
|---------------------------------------------------------------------------------------------------------------------------------------------------|
+| `hosts` | Yes | | ES address, can be one
or multiple addresses, or the load balancer address of ES
|
+| `user` | No | Empty | ES username
|
+| `password` | No | Empty | Password of the
corresponding user
|
+| `doc_value_scan` | No | true | Whether to obtain
value of the target field by ES/Lucene columnar storage
|
+| `keyword_sniff` | No | true | Whether to sniff the
text.fields in ES based on keyword; If this is set to false, the system will
perform matching after tokenization. |
| `nodes_discovery` | No | true | Whether to enable ES
node discovery, set to true by default; set to false in network isolation
environments and only connected to specified nodes |
-| `ssl` | No | false | Whether to enable
HTTPS access mode for ES, currently follows a "Trust All" method in FE/BE |
-| `mapping_es_id` | No | false | Whether to map the
`_id` field in the ES index |
+| `ssl` | No | false | Whether to enable
HTTPS access mode for ES, currently follows a "Trust All" method in FE/BE
|
+| `mapping_es_id` | No | false | Whether to map the
`_id` field in the ES index
|
+| `like_push_down` | No | true | Whether to transform
like to wildcard push down to es, this increases the cpu consumption of the es.
|
> 1. In terms of authentication, only HTTP Basic authentication is supported
> and it requires the user to have read privilege for the index and paths
> including `/_cluster/state/` and `_nodes/http` ; if you have not enabled
> security authentication for the cluster, you don't need to set the `user`
> and `password`.
>
diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/es.md
b/docs/zh-CN/docs/lakehouse/multi-catalog/es.md
index 68fea116ba..c6225d41b0 100644
--- a/docs/zh-CN/docs/lakehouse/multi-catalog/es.md
+++ b/docs/zh-CN/docs/lakehouse/multi-catalog/es.md
@@ -50,16 +50,17 @@ CREATE CATALOG es PROPERTIES (
### 参数说明
-参数 | 是否必须 | 默认值 | 说明
---- | --- | --- | ---
-`hosts` | 是 | | ES 地址，可以是一个或多个，也可以是 ES 的负载均衡地址 |
-`user` | 否 | 空 | ES 用户名 |
-`password` | 否 | 空 | 对应用户的密码信息 |
-`doc_value_scan` | 否 | true | 是否开启通过 ES/Lucene 列式存储获取查询字段的值 |
+参数 | 是否必须 | 默认值 | 说明
+--- | --- | ---
|------------------------------------------------------------------------
+`hosts` | 是 | | ES 地址，可以是一个或多个，也可以是 ES 的负载均衡地址
|
+`user` | 否 | 空 | ES 用户名
|
+`password` | 否 | 空 | 对应用户的密码信息
|
+`doc_value_scan` | 否 | true | 是否开启通过 ES/Lucene 列式存储获取查询字段的值
|
`keyword_sniff` | 否 | true | 是否对 ES 中字符串分词类型 text.fields 进行探测，通过 keyword
进行查询。设置为 false 会按照分词后的内容匹配 |
-`nodes_discovery` | 否 | true | 是否开启 ES 节点发现，默认为 true，在网络隔离环境下设置为 false，只连接指定节点
|
-`ssl` | 否 | false | ES 是否开启 https 访问模式，目前在 fe/be 实现方式为信任所有 |
-`mapping_es_id` | 否 | false | 是否映射 ES 索引中的 `_id` 字段 |
+`nodes_discovery` | 否 | true | 是否开启 ES 节点发现，默认为 true，在网络隔离环境下设置为 false，只连接指定节点
|
+`ssl` | 否 | false | ES 是否开启 https 访问模式，目前在 fe/be 实现方式为信任所有
|
+`mapping_es_id` | 否 | false | 是否映射 ES 索引中的 `_id` 字段
|
+`like_push_down` | 否 | true | 是否将 like 转化为 wildchard 下推到 ES，会增加 ES cpu 消耗
|
> 1. 认证方式目前仅支持 Http Basic 认证，并且需要确保该用户有访问: `/_cluster/state/、_nodes/http` 等路径和
> index 的读权限; 集群未开启安全认证，用户名和密码不需要设置。
>
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
index 6e27a626f2..4e1f10dd3f 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
@@ -3042,7 +3042,8 @@ public class Env {
sb.append("\"max_docvalue_fields\" =
\"").append(esTable.getMaxDocValueFields()).append("\",\n");
sb.append("\"enable_keyword_sniff\" =
\"").append(esTable.isEnableKeywordSniff()).append("\",\n");
sb.append("\"nodes_discovery\" =
\"").append(esTable.isNodesDiscovery()).append("\",\n");
- sb.append("\"http_ssl_enabled\" =
\"").append(esTable.isHttpSslEnabled()).append("\"\n");
+ sb.append("\"http_ssl_enabled\" =
\"").append(esTable.isHttpSslEnabled()).append("\",\n");
+ sb.append("\"like_push_down\" =
\"").append(esTable.isLikePushDown()).append("\"\n");
sb.append(")");
} else if (table.getType() == TableType.HIVE) {
HiveTable hiveTable = (HiveTable) table;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/EsResource.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/EsResource.java
index 012ad119e3..91cb4529ad 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/EsResource.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/EsResource.java
@@ -60,6 +60,8 @@ public class EsResource extends Resource {
public static final String NODES_DISCOVERY = "nodes_discovery";
public static final String HTTP_SSL_ENABLED = "http_ssl_enabled";
public static final String MAPPING_ES_ID = "mapping_es_id";
+
+ public static final String LIKE_PUSH_DOWN = "like_push_down";
public static final String QUERY_DSL = "query_dsl";
public static final String DOC_VALUE_SCAN_DEFAULT_VALUE = "true";
@@ -67,6 +69,8 @@ public class EsResource extends Resource {
public static final String HTTP_SSL_ENABLED_DEFAULT_VALUE = "false";
public static final String NODES_DISCOVERY_DEFAULT_VALUE = "true";
public static final String MAPPING_ES_ID_DEFAULT_VALUE = "false";
+
+ public static final String LIKE_PUSH_DOWN_DEFAULT_VALUE = "true";
@SerializedName(value = "properties")
private Map<String, String> properties;
@@ -127,6 +131,9 @@ public class EsResource extends Resource {
if (properties.containsKey(EsResource.MAPPING_ES_ID)) {
EsUtil.getBoolean(properties, EsResource.MAPPING_ES_ID);
}
+ if (properties.containsKey(EsResource.LIKE_PUSH_DOWN)) {
+ EsUtil.getBoolean(properties, EsResource.LIKE_PUSH_DOWN);
+ }
}
private Map<String, String> processCompatibleProperties(Map<String,
String> props) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/EsTable.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/EsTable.java
index a26a7db7f9..81000c69d1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/EsTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/EsTable.java
@@ -87,10 +87,15 @@ public class EsTable extends Table {
// would downgrade to extract value from `stored_fields`
private int maxDocValueFields = DEFAULT_MAX_DOCVALUE_FIELDS;
+ // Whether to enable the discovery of es nodes, You can disable it if you
are in network isolation
private boolean nodesDiscovery =
Boolean.parseBoolean(EsResource.NODES_DISCOVERY_DEFAULT_VALUE);
+ // Whether to use ssl call es, be and fe access through trust
private boolean httpSslEnabled =
Boolean.parseBoolean(EsResource.HTTP_SSL_ENABLED_DEFAULT_VALUE);
+ // Whether pushdown like expr, like will trans to wildcard query, consumes
too many es cpu resources
+ private boolean likePushDown =
Boolean.parseBoolean(EsResource.LIKE_PUSH_DOWN_DEFAULT_VALUE);
+
// tableContext is used for being convenient to persist some configuration
parameters uniformly
private Map<String, String> tableContext = new HashMap<>();
@@ -171,6 +176,10 @@ public class EsTable extends Table {
httpSslEnabled = EsUtil.getBoolean(properties,
EsResource.HTTP_SSL_ENABLED);
}
+ if (properties.containsKey(EsResource.LIKE_PUSH_DOWN)) {
+ likePushDown = EsUtil.getBoolean(properties,
EsResource.LIKE_PUSH_DOWN);
+ }
+
if (StringUtils.isNotBlank(properties.get(EsResource.TYPE))) {
mappingType = properties.get(EsResource.TYPE).trim();
}
@@ -197,6 +206,7 @@ public class EsTable extends Table {
tableContext.put("maxDocValueFields",
String.valueOf(maxDocValueFields));
tableContext.put(EsResource.NODES_DISCOVERY,
String.valueOf(nodesDiscovery));
tableContext.put(EsResource.HTTP_SSL_ENABLED,
String.valueOf(httpSslEnabled));
+ tableContext.put(EsResource.LIKE_PUSH_DOWN,
String.valueOf(likePushDown));
}
@Override
@@ -275,6 +285,8 @@ public class EsTable extends Table {
EsResource.NODES_DISCOVERY_DEFAULT_VALUE));
httpSslEnabled =
Boolean.parseBoolean(tableContext.getOrDefault(EsResource.HTTP_SSL_ENABLED,
EsResource.HTTP_SSL_ENABLED_DEFAULT_VALUE));
+ likePushDown =
Boolean.parseBoolean(tableContext.getOrDefault(EsResource.LIKE_PUSH_DOWN,
+ EsResource.LIKE_PUSH_DOWN_DEFAULT_VALUE));
PartitionType partType = PartitionType.valueOf(Text.readString(in));
if (partType == PartitionType.UNPARTITIONED) {
partitionInfo = SinglePartitionInfo.read(in);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/external/EsExternalTable.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/external/EsExternalTable.java
index 4eab32b17a..290917e21c 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/catalog/external/EsExternalTable.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/catalog/external/EsExternalTable.java
@@ -96,6 +96,7 @@ public class EsExternalTable extends ExternalTable {
esTable.setEnableKeywordSniff(esCatalog.enableKeywordSniff());
esTable.setNodesDiscovery(esCatalog.enableNodesDiscovery());
esTable.setHttpSslEnabled(esCatalog.enableSsl());
+ esTable.setLikePushDown(esCatalog.enableLikePushDown());
esTable.setSeeds(esCatalog.getNodes());
esTable.setHosts(String.join(",", esCatalog.getNodes()));
esTable.syncTableMetaData();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalCatalog.java
index 390963ae58..650e5374e5 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalCatalog.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalCatalog.java
@@ -106,6 +106,11 @@ public class EsExternalCatalog extends ExternalCatalog {
EsResource.MAPPING_ES_ID_DEFAULT_VALUE));
}
+ public boolean enableLikePushDown() {
+ return
Boolean.parseBoolean(catalogProperty.getOrDefault(EsResource.LIKE_PUSH_DOWN,
+ EsResource.LIKE_PUSH_DOWN_DEFAULT_VALUE));
+ }
+
@Override
protected void initLocalObjectsImpl() {
esRestClient = new EsRestClient(getNodes(), getUsername(),
getPassword(), enableSsl());
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java
b/fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java
index 2276356fa8..4dd1991853 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/EsUtil.java
@@ -17,32 +17,15 @@
package org.apache.doris.external.elasticsearch;
-import org.apache.doris.analysis.BinaryPredicate;
-import org.apache.doris.analysis.BoolLiteral;
-import org.apache.doris.analysis.CastExpr;
-import org.apache.doris.analysis.CompoundPredicate;
-import org.apache.doris.analysis.DecimalLiteral;
import org.apache.doris.analysis.DistributionDesc;
-import org.apache.doris.analysis.Expr;
-import org.apache.doris.analysis.FloatLiteral;
-import org.apache.doris.analysis.FunctionCallExpr;
-import org.apache.doris.analysis.InPredicate;
-import org.apache.doris.analysis.IntLiteral;
-import org.apache.doris.analysis.IsNullPredicate;
-import org.apache.doris.analysis.LargeIntLiteral;
-import org.apache.doris.analysis.LikePredicate;
-import org.apache.doris.analysis.LikePredicate.Operator;
import org.apache.doris.analysis.PartitionDesc;
import org.apache.doris.analysis.RangePartitionDesc;
-import org.apache.doris.analysis.SlotRef;
import org.apache.doris.catalog.ArrayType;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.ScalarType;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.DdlException;
-import org.apache.doris.external.elasticsearch.QueryBuilders.QueryBuilder;
-import org.apache.doris.thrift.TExprOpcode;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -50,12 +33,10 @@ import org.json.simple.JSONObject;
import org.json.simple.JSONValue;
import java.util.ArrayList;
-import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Set;
-import java.util.stream.Collectors;
/**
* Util for ES, some static method.
@@ -213,167 +194,6 @@ public class EsUtil {
return properties;
}
- private static QueryBuilder toCompoundEsDsl(Expr expr, List<Expr>
notPushDownList,
- Map<String, String> fieldsContext) {
- CompoundPredicate compoundPredicate = (CompoundPredicate) expr;
- switch (compoundPredicate.getOp()) {
- case AND: {
- QueryBuilder left = toEsDsl(compoundPredicate.getChild(0),
notPushDownList, fieldsContext);
- QueryBuilder right = toEsDsl(compoundPredicate.getChild(1),
notPushDownList, fieldsContext);
- if (left != null && right != null) {
- return QueryBuilders.boolQuery().must(left).must(right);
- }
- return null;
- }
- case OR: {
- int beforeSize = notPushDownList.size();
- QueryBuilder left = toEsDsl(compoundPredicate.getChild(0),
notPushDownList, fieldsContext);
- QueryBuilder right = toEsDsl(compoundPredicate.getChild(1),
notPushDownList, fieldsContext);
- int afterSize = notPushDownList.size();
- if (left != null && right != null) {
- return
QueryBuilders.boolQuery().should(left).should(right);
- }
- // One 'or' association cannot be pushed down and the other
cannot be pushed down
- if (afterSize > beforeSize) {
- if (left != null) {
- // add right if right don't pushdown
- notPushDownList.add(compoundPredicate.getChild(0));
- } else if (right != null) {
- // add left if left don't pushdown
- notPushDownList.add(compoundPredicate.getChild(1));
- }
- }
- return null;
- }
- case NOT: {
- QueryBuilder child = toEsDsl(compoundPredicate.getChild(0),
notPushDownList, fieldsContext);
- if (child != null) {
- return QueryBuilders.boolQuery().mustNot(child);
- }
- return null;
- }
- default:
- return null;
- }
- }
-
- private static Expr exprWithoutCast(Expr expr) {
- if (expr instanceof CastExpr) {
- return exprWithoutCast(expr.getChild(0));
- }
- return expr;
- }
-
- public static QueryBuilder toEsDsl(Expr expr) {
- return toEsDsl(expr, new ArrayList<>(), new HashMap<>());
- }
-
- /**
- * Doris expr to es dsl.
- **/
- public static QueryBuilder toEsDsl(Expr expr, List<Expr> notPushDownList,
Map<String, String> fieldsContext) {
- if (expr == null) {
- return null;
- }
- // CompoundPredicate, `between` also converted to CompoundPredicate.
- if (expr instanceof CompoundPredicate) {
- return toCompoundEsDsl(expr, notPushDownList, fieldsContext);
- }
- TExprOpcode opCode = expr.getOpcode();
- String column;
- Expr leftExpr = expr.getChild(0);
- // Type transformed cast can not pushdown
- if (leftExpr instanceof CastExpr) {
- Expr withoutCastExpr = exprWithoutCast(leftExpr);
- // pushdown col(float) >= 3
- if (withoutCastExpr.getType().equals(leftExpr.getType()) ||
(withoutCastExpr.getType().isFloatingPointType()
- && leftExpr.getType().isFloatingPointType())) {
- column = ((SlotRef) withoutCastExpr).getColumnName();
- } else {
- notPushDownList.add(expr);
- return null;
- }
- } else if (leftExpr instanceof SlotRef) {
- column = ((SlotRef) leftExpr).getColumnName();
- } else {
- notPushDownList.add(expr);
- return null;
- }
- // Replace col with col.keyword if mapping exist.
- column = fieldsContext.getOrDefault(column, column);
- if (expr instanceof BinaryPredicate) {
- Object value = toDorisLiteral(expr.getChild(1));
- switch (opCode) {
- case EQ:
- case EQ_FOR_NULL:
- return QueryBuilders.termQuery(column, value);
- case NE:
- return
QueryBuilders.boolQuery().mustNot(QueryBuilders.termQuery(column, value));
- case GE:
- return QueryBuilders.rangeQuery(column).gte(value);
- case GT:
- return QueryBuilders.rangeQuery(column).gt(value);
- case LE:
- return QueryBuilders.rangeQuery(column).lte(value);
- case LT:
- return QueryBuilders.rangeQuery(column).lt(value);
- default:
- return null;
- }
- }
- if (expr instanceof IsNullPredicate) {
- IsNullPredicate isNullPredicate = (IsNullPredicate) expr;
- if (isNullPredicate.isNotNull()) {
- return QueryBuilders.existsQuery(column);
- }
- return
QueryBuilders.boolQuery().mustNot(QueryBuilders.existsQuery(column));
- }
- if (expr instanceof LikePredicate) {
- LikePredicate likePredicate = (LikePredicate) expr;
- if (likePredicate.getOp().equals(Operator.LIKE)) {
- char[] chars =
likePredicate.getChild(1).getStringValue().toCharArray();
- // example of translation :
- // abc_123 ===> abc?123
- // abc%ykz ===> abc*123
- // %abc123 ===> *abc123
- // _abc123 ===> ?abc123
- // \\_abc1 ===> \\_abc1
- // abc\\_123 ===> abc\\_123
- // abc\\%123 ===> abc\\%123
- // NOTE. user must input sql like 'abc\\_123' or 'abc\\%ykz'
- for (int i = 0; i < chars.length; i++) {
- if (chars[i] == '_' || chars[i] == '%') {
- if (i == 0) {
- chars[i] = (chars[i] == '_') ? '?' : '*';
- } else if (chars[i - 1] != '\\') {
- chars[i] = (chars[i] == '_') ? '?' : '*';
- }
- }
- }
- return QueryBuilders.wildcardQuery(column, new String(chars));
- } else {
- return QueryBuilders.wildcardQuery(column,
likePredicate.getChild(1).getStringValue());
- }
- }
- if (expr instanceof InPredicate) {
- InPredicate inPredicate = (InPredicate) expr;
- List<Object> values =
inPredicate.getListChildren().stream().map(EsUtil::toDorisLiteral)
- .collect(Collectors.toList());
- if (inPredicate.isNotIn()) {
- return
QueryBuilders.boolQuery().mustNot(QueryBuilders.termsQuery(column, values));
- }
- return QueryBuilders.termsQuery(column, values);
- }
- if (expr instanceof FunctionCallExpr) {
- FunctionCallExpr functionCallExpr = (FunctionCallExpr) expr;
- if ("esquery".equals(functionCallExpr.getFnName().getFunction())) {
- String stringValue =
functionCallExpr.getChild(1).getStringValue();
- return new QueryBuilders.EsQueryBuilder(stringValue);
- }
- }
- return null;
- }
-
/**
* Generate columns from ES Cluster.
@@ -458,28 +278,5 @@ public class EsUtil {
}
}
- private static Object toDorisLiteral(Expr expr) {
- if (!expr.isLiteral()) {
- return null;
- }
- if (expr instanceof BoolLiteral) {
- BoolLiteral boolLiteral = (BoolLiteral) expr;
- return boolLiteral.getValue();
- } else if (expr instanceof DecimalLiteral) {
- DecimalLiteral decimalLiteral = (DecimalLiteral) expr;
- return decimalLiteral.getValue();
- } else if (expr instanceof FloatLiteral) {
- FloatLiteral floatLiteral = (FloatLiteral) expr;
- return floatLiteral.getValue();
- } else if (expr instanceof IntLiteral) {
- IntLiteral intLiteral = (IntLiteral) expr;
- return intLiteral.getValue();
- } else if (expr instanceof LargeIntLiteral) {
- LargeIntLiteral largeIntLiteral = (LargeIntLiteral) expr;
- return largeIntLiteral.getLongValue();
- }
- return expr.getStringValue();
- }
-
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/QueryBuilders.java
b/fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/QueryBuilders.java
index 3dd800cb91..cc269a452a 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/QueryBuilders.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/external/elasticsearch/QueryBuilders.java
@@ -17,19 +17,42 @@
package org.apache.doris.external.elasticsearch;
+import org.apache.doris.analysis.BinaryPredicate;
+import org.apache.doris.analysis.BoolLiteral;
+import org.apache.doris.analysis.CastExpr;
+import org.apache.doris.analysis.CompoundPredicate;
+import org.apache.doris.analysis.DecimalLiteral;
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.FloatLiteral;
+import org.apache.doris.analysis.FunctionCallExpr;
+import org.apache.doris.analysis.InPredicate;
+import org.apache.doris.analysis.IntLiteral;
+import org.apache.doris.analysis.IsNullPredicate;
+import org.apache.doris.analysis.LargeIntLiteral;
+import org.apache.doris.analysis.LikePredicate;
+import org.apache.doris.analysis.LikePredicate.Operator;
+import org.apache.doris.analysis.SlotRef;
+import org.apache.doris.catalog.EsResource;
+import org.apache.doris.thrift.TExprOpcode;
+
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
+import lombok.Builder;
+import lombok.Data;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import java.io.IOException;
import java.io.StringWriter;
import java.util.ArrayList;
+import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
+import java.util.Map;
import java.util.Map.Entry;
import java.util.Objects;
+import java.util.stream.Collectors;
/**
@@ -38,6 +61,233 @@ import java.util.Objects;
*/
public final class QueryBuilders {
+ /**
+ * Generate dsl from compound expr.
+ **/
+ private static QueryBuilder toCompoundEsDsl(Expr expr, List<Expr>
notPushDownList,
+ Map<String, String> fieldsContext, BuilderOptions builderOptions) {
+ CompoundPredicate compoundPredicate = (CompoundPredicate) expr;
+ switch (compoundPredicate.getOp()) {
+ case AND: {
+ QueryBuilder left = toEsDsl(compoundPredicate.getChild(0),
notPushDownList, fieldsContext,
+ builderOptions);
+ QueryBuilder right = toEsDsl(compoundPredicate.getChild(1),
notPushDownList, fieldsContext,
+ builderOptions);
+ if (left != null && right != null) {
+ return QueryBuilders.boolQuery().must(left).must(right);
+ }
+ return null;
+ }
+ case OR: {
+ int beforeSize = notPushDownList.size();
+ QueryBuilder left = toEsDsl(compoundPredicate.getChild(0),
notPushDownList, fieldsContext,
+ builderOptions);
+ QueryBuilder right = toEsDsl(compoundPredicate.getChild(1),
notPushDownList, fieldsContext,
+ builderOptions);
+ int afterSize = notPushDownList.size();
+ if (left != null && right != null) {
+ return
QueryBuilders.boolQuery().should(left).should(right);
+ }
+ // One 'or' association cannot be pushed down and the other
cannot be pushed down
+ if (afterSize > beforeSize) {
+ notPushDownList.add(compoundPredicate);
+ }
+ return null;
+ }
+ case NOT: {
+ QueryBuilder child = toEsDsl(compoundPredicate.getChild(0),
notPushDownList, fieldsContext,
+ builderOptions);
+ if (child != null) {
+ return QueryBuilders.boolQuery().mustNot(child);
+ }
+ return null;
+ }
+ default:
+ return null;
+ }
+ }
+
+ /**
+ * Get the expr inside the cast.
+ **/
+ private static Expr exprWithoutCast(Expr expr) {
+ if (expr instanceof CastExpr) {
+ return exprWithoutCast(expr.getChild(0));
+ }
+ return expr;
+ }
+
+ public static QueryBuilder toEsDsl(Expr expr) {
+ return toEsDsl(expr, new ArrayList<>(), new HashMap<>(),
+
BuilderOptions.builder().likePushDown(Boolean.parseBoolean(EsResource.LIKE_PUSH_DOWN_DEFAULT_VALUE))
+ .build());
+ }
+
+ private static QueryBuilder parseBinaryPredicate(Expr expr, TExprOpcode
opCode, String column) {
+ Object value = toDorisLiteral(expr.getChild(1));
+ switch (opCode) {
+ case EQ:
+ case EQ_FOR_NULL:
+ return QueryBuilders.termQuery(column, value);
+ case NE:
+ return
QueryBuilders.boolQuery().mustNot(QueryBuilders.termQuery(column, value));
+ case GE:
+ return QueryBuilders.rangeQuery(column).gte(value);
+ case GT:
+ return QueryBuilders.rangeQuery(column).gt(value);
+ case LE:
+ return QueryBuilders.rangeQuery(column).lte(value);
+ case LT:
+ return QueryBuilders.rangeQuery(column).lt(value);
+ default:
+ return null;
+ }
+ }
+
+ private static QueryBuilder parseIsNullPredicate(Expr expr, String column)
{
+ IsNullPredicate isNullPredicate = (IsNullPredicate) expr;
+ if (isNullPredicate.isNotNull()) {
+ return QueryBuilders.existsQuery(column);
+ }
+ return
QueryBuilders.boolQuery().mustNot(QueryBuilders.existsQuery(column));
+ }
+
+ private static QueryBuilder parseLikePredicate(Expr expr, String column) {
+ LikePredicate likePredicate = (LikePredicate) expr;
+ if (likePredicate.getOp().equals(Operator.LIKE)) {
+ char[] chars =
likePredicate.getChild(1).getStringValue().toCharArray();
+ // example of translation :
+ // abc_123 ===> abc?123
+ // abc%ykz ===> abc*123
+ // %abc123 ===> *abc123
+ // _abc123 ===> ?abc123
+ // \\_abc1 ===> \\_abc1
+ // abc\\_123 ===> abc\\_123
+ // abc\\%123 ===> abc\\%123
+ // NOTE. user must input sql like 'abc\\_123' or 'abc\\%ykz'
+ for (int i = 0; i < chars.length; i++) {
+ if (chars[i] == '_' || chars[i] == '%') {
+ if (i == 0) {
+ chars[i] = (chars[i] == '_') ? '?' : '*';
+ } else if (chars[i - 1] != '\\') {
+ chars[i] = (chars[i] == '_') ? '?' : '*';
+ }
+ }
+ }
+ return QueryBuilders.wildcardQuery(column, new String(chars));
+ } else {
+ return QueryBuilders.wildcardQuery(column,
likePredicate.getChild(1).getStringValue());
+ }
+ }
+
+ private static QueryBuilder parseInPredicate(Expr expr, String column) {
+ InPredicate inPredicate = (InPredicate) expr;
+ List<Object> values =
inPredicate.getListChildren().stream().map(QueryBuilders::toDorisLiteral)
+ .collect(Collectors.toList());
+ if (inPredicate.isNotIn()) {
+ return
QueryBuilders.boolQuery().mustNot(QueryBuilders.termsQuery(column, values));
+ }
+ return QueryBuilders.termsQuery(column, values);
+ }
+
+ private static QueryBuilder parseFunctionCallExpr(Expr expr) {
+ // esquery(k1, '{
+ // "match_phrase": {
+ // "k1": "doris on es"
+ // }
+ // }');
+ // The first child k1 compatible with expr syntax
+ FunctionCallExpr functionCallExpr = (FunctionCallExpr) expr;
+ if ("esquery".equals(functionCallExpr.getFnName().getFunction())) {
+ String stringValue = functionCallExpr.getChild(1).getStringValue();
+ return new QueryBuilders.EsQueryBuilder(stringValue);
+ }
+ return null;
+ }
+
+ /**
+ * Doris expr to es dsl.
+ **/
+ public static QueryBuilder toEsDsl(Expr expr, List<Expr> notPushDownList,
Map<String, String> fieldsContext,
+ BuilderOptions builderOptions) {
+ if (expr == null) {
+ return null;
+ }
+ // CompoundPredicate, `between` also converted to CompoundPredicate.
+ if (expr instanceof CompoundPredicate) {
+ return toCompoundEsDsl(expr, notPushDownList, fieldsContext,
builderOptions);
+ }
+ TExprOpcode opCode = expr.getOpcode();
+ String column;
+ Expr leftExpr = expr.getChild(0);
+ // Type transformed cast can not pushdown
+ if (leftExpr instanceof CastExpr) {
+ Expr withoutCastExpr = exprWithoutCast(leftExpr);
+ // pushdown col(float) >= 3
+ if (withoutCastExpr.getType().equals(leftExpr.getType()) ||
(withoutCastExpr.getType().isFloatingPointType()
+ && leftExpr.getType().isFloatingPointType())) {
+ column = ((SlotRef) withoutCastExpr).getColumnName();
+ } else {
+ notPushDownList.add(expr);
+ return null;
+ }
+ } else if (leftExpr instanceof SlotRef) {
+ column = ((SlotRef) leftExpr).getColumnName();
+ } else {
+ notPushDownList.add(expr);
+ return null;
+ }
+ // Replace col with col.keyword if mapping exist.
+ column = fieldsContext.getOrDefault(column, column);
+ if (expr instanceof BinaryPredicate) {
+ return parseBinaryPredicate(expr, opCode, column);
+ }
+ if (expr instanceof IsNullPredicate) {
+ return parseIsNullPredicate(expr, column);
+ }
+ if (expr instanceof LikePredicate) {
+ if (!builderOptions.isLikePushDown()) {
+ notPushDownList.add(expr);
+ return null;
+ } else {
+ return parseLikePredicate(expr, column);
+ }
+ }
+ if (expr instanceof InPredicate) {
+ return parseInPredicate(expr, column);
+ }
+ if (expr instanceof FunctionCallExpr) {
+ return parseFunctionCallExpr(expr);
+ }
+ return null;
+ }
+
+ /**
+ * Expr trans to doris literal.
+ **/
+ private static Object toDorisLiteral(Expr expr) {
+ if (!expr.isLiteral()) {
+ return null;
+ }
+ if (expr instanceof BoolLiteral) {
+ BoolLiteral boolLiteral = (BoolLiteral) expr;
+ return boolLiteral.getValue();
+ } else if (expr instanceof DecimalLiteral) {
+ DecimalLiteral decimalLiteral = (DecimalLiteral) expr;
+ return decimalLiteral.getValue();
+ } else if (expr instanceof FloatLiteral) {
+ FloatLiteral floatLiteral = (FloatLiteral) expr;
+ return floatLiteral.getValue();
+ } else if (expr instanceof IntLiteral) {
+ IntLiteral intLiteral = (IntLiteral) expr;
+ return intLiteral.getValue();
+ } else if (expr instanceof LargeIntLiteral) {
+ LargeIntLiteral largeIntLiteral = (LargeIntLiteral) expr;
+ return largeIntLiteral.getLongValue();
+ }
+ return expr.getStringValue();
+ }
+
/**
* A query that matches on all documents.
*/
@@ -48,7 +298,7 @@ public final class QueryBuilders {
/**
* A Query that matches documents containing a term.
*
- * @param name The name of the field
+ * @param name The name of the field
* @param value The value of the term
*/
public static TermQueryBuilder termQuery(String name, String value) {
@@ -58,7 +308,7 @@ public final class QueryBuilders {
/**
* A Query that matches documents containing a term.
*
- * @param name The name of the field
+ * @param name The name of the field
* @param value The value of the term
*/
public static TermQueryBuilder termQuery(String name, int value) {
@@ -68,7 +318,7 @@ public final class QueryBuilders {
/**
* A Query that matches documents containing a term.
*
- * @param name The name of the field
+ * @param name The name of the field
* @param value The value of the term
*/
public static TermQueryBuilder termQuery(String name, long value) {
@@ -78,7 +328,7 @@ public final class QueryBuilders {
/**
* A Query that matches documents containing a term.
*
- * @param name The name of the field
+ * @param name The name of the field
* @param value The value of the term
*/
public static TermQueryBuilder termQuery(String name, float value) {
@@ -88,7 +338,7 @@ public final class QueryBuilders {
/**
* A Query that matches documents containing a term.
*
- * @param name The name of the field
+ * @param name The name of the field
* @param value The value of the term
*/
public static TermQueryBuilder termQuery(String name, double value) {
@@ -98,7 +348,7 @@ public final class QueryBuilders {
/**
* A Query that matches documents containing a term.
*
- * @param name The name of the field
+ * @param name The name of the field
* @param value The value of the term
*/
public static TermQueryBuilder termQuery(String name, boolean value) {
@@ -108,7 +358,7 @@ public final class QueryBuilders {
/**
* A Query that matches documents containing a term.
*
- * @param name The name of the field
+ * @param name The name of the field
* @param value The value of the term
*/
public static TermQueryBuilder termQuery(String name, Object value) {
@@ -123,7 +373,7 @@ public final class QueryBuilders {
* a Wildcard term should not start with one of the wildcards {@code *} or
* {@code ?}.
*
- * @param name The field name
+ * @param name The field name
* @param query The wildcard query string
*/
public static WildcardQueryBuilder wildcardQuery(String name, String
query) {
@@ -141,7 +391,7 @@ public final class QueryBuilders {
/**
* A filter for a field based on several terms matching on any of them.
*
- * @param name The field name
+ * @param name The field name
* @param values The terms
*/
public static TermsQueryBuilder termsQuery(String name, Iterable<?>
values) {
@@ -166,6 +416,16 @@ public final class QueryBuilders {
return new RangeQueryBuilder(name);
}
+ /**
+ * Used to pass some parameters to generate the dsl
+ **/
+ @Builder
+ @Data
+ public static class BuilderOptions {
+
+ private boolean likePushDown;
+ }
+
/**
* Base class to build various ES queries
@@ -505,7 +765,7 @@ public final class QueryBuilders {
/**
* Write (scalar) value (string, number, boolean or null) to json format
*
- * @param out source target
+ * @param out source target
* @param value value to write
* @throws IOException if error
*/
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/EsScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/EsScanNode.java
index 5a6667eb71..7c9aae4a8d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/EsScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/EsScanNode.java
@@ -33,9 +33,9 @@ import org.apache.doris.common.UserException;
import org.apache.doris.external.elasticsearch.EsShardPartitions;
import org.apache.doris.external.elasticsearch.EsShardRouting;
import org.apache.doris.external.elasticsearch.EsTablePartitions;
-import org.apache.doris.external.elasticsearch.EsUtil;
import org.apache.doris.external.elasticsearch.QueryBuilders;
import org.apache.doris.external.elasticsearch.QueryBuilders.BoolQueryBuilder;
+import org.apache.doris.external.elasticsearch.QueryBuilders.BuilderOptions;
import org.apache.doris.external.elasticsearch.QueryBuilders.QueryBuilder;
import org.apache.doris.statistics.StatisticalType;
import org.apache.doris.system.Backend;
@@ -362,7 +362,8 @@ public class EsScanNode extends ScanNode {
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
List<Expr> notPushDownList = new ArrayList<>();
for (Expr expr : conjuncts) {
- QueryBuilder queryBuilder = EsUtil.toEsDsl(expr,
notPushDownList, fieldsContext);
+ QueryBuilder queryBuilder = QueryBuilders.toEsDsl(expr,
notPushDownList, fieldsContext,
+
BuilderOptions.builder().likePushDown(table.isLikePushDown()).build());
if (queryBuilder != null) {
hasFilter = true;
boolQueryBuilder.must(queryBuilder);
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/external/elasticsearch/EsUtilTest.java
b/fe/fe-core/src/test/java/org/apache/doris/external/elasticsearch/EsUtilTest.java
index 7783b4d437..a744b0fba0 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/external/elasticsearch/EsUtilTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/external/elasticsearch/EsUtilTest.java
@@ -31,10 +31,12 @@ import org.apache.doris.analysis.LikePredicate;
import org.apache.doris.analysis.SlotRef;
import org.apache.doris.analysis.StringLiteral;
import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.EsResource;
import org.apache.doris.catalog.EsTable;
import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.ExceptionChecker;
+import org.apache.doris.external.elasticsearch.QueryBuilders.BuilderOptions;
import mockit.Expectations;
import mockit.Injectable;
@@ -179,13 +181,14 @@ public class EsUtilTest extends EsTestCase {
Expr ltExpr = new BinaryPredicate(Operator.LT, k1, intLiteral);
Expr gtExpr = new BinaryPredicate(Operator.GT, k1, intLiteral);
Expr efnExpr = new BinaryPredicate(Operator.EQ_FOR_NULL, new
SlotRef(null, "k1"), new IntLiteral(3));
- Assertions.assertEquals("{\"term\":{\"k1\":3}}",
EsUtil.toEsDsl(eqExpr).toJson());
-
Assertions.assertEquals("{\"bool\":{\"must_not\":{\"term\":{\"k1\":3}}}}",
EsUtil.toEsDsl(neExpr).toJson());
- Assertions.assertEquals("{\"range\":{\"k1\":{\"lte\":3}}}",
EsUtil.toEsDsl(leExpr).toJson());
- Assertions.assertEquals("{\"range\":{\"k1\":{\"gte\":3}}}",
EsUtil.toEsDsl(geExpr).toJson());
- Assertions.assertEquals("{\"range\":{\"k1\":{\"lt\":3}}}",
EsUtil.toEsDsl(ltExpr).toJson());
- Assertions.assertEquals("{\"range\":{\"k1\":{\"gt\":3}}}",
EsUtil.toEsDsl(gtExpr).toJson());
- Assertions.assertEquals("{\"term\":{\"k1\":3}}",
EsUtil.toEsDsl(efnExpr).toJson());
+ Assertions.assertEquals("{\"term\":{\"k1\":3}}",
QueryBuilders.toEsDsl(eqExpr).toJson());
+
Assertions.assertEquals("{\"bool\":{\"must_not\":{\"term\":{\"k1\":3}}}}",
+ QueryBuilders.toEsDsl(neExpr).toJson());
+ Assertions.assertEquals("{\"range\":{\"k1\":{\"lte\":3}}}",
QueryBuilders.toEsDsl(leExpr).toJson());
+ Assertions.assertEquals("{\"range\":{\"k1\":{\"gte\":3}}}",
QueryBuilders.toEsDsl(geExpr).toJson());
+ Assertions.assertEquals("{\"range\":{\"k1\":{\"lt\":3}}}",
QueryBuilders.toEsDsl(ltExpr).toJson());
+ Assertions.assertEquals("{\"range\":{\"k1\":{\"gt\":3}}}",
QueryBuilders.toEsDsl(gtExpr).toJson());
+ Assertions.assertEquals("{\"term\":{\"k1\":3}}",
QueryBuilders.toEsDsl(efnExpr).toJson());
}
@Test
@@ -202,11 +205,11 @@ public class EsUtilTest extends EsTestCase {
binaryPredicate2);
CompoundPredicate notPredicate = new
CompoundPredicate(CompoundPredicate.Operator.NOT, binaryPredicate1, null);
Assertions.assertEquals("{\"bool\":{\"must\":[{\"term\":{\"k1\":3}},{\"range\":{\"k2\":{\"gt\":5}}}]}}",
- EsUtil.toEsDsl(andPredicate).toJson());
+ QueryBuilders.toEsDsl(andPredicate).toJson());
Assertions.assertEquals("{\"bool\":{\"should\":[{\"term\":{\"k1\":3}},{\"range\":{\"k2\":{\"gt\":5}}}]}}",
- EsUtil.toEsDsl(orPredicate).toJson());
+ QueryBuilders.toEsDsl(orPredicate).toJson());
Assertions.assertEquals("{\"bool\":{\"must_not\":{\"term\":{\"k1\":3}}}}",
- EsUtil.toEsDsl(notPredicate).toJson());
+ QueryBuilders.toEsDsl(notPredicate).toJson());
}
@Test
@@ -215,8 +218,8 @@ public class EsUtilTest extends EsTestCase {
IsNullPredicate isNullPredicate = new IsNullPredicate(k1, false);
IsNullPredicate isNotNullPredicate = new IsNullPredicate(k1, true);
Assertions.assertEquals("{\"bool\":{\"must_not\":{\"exists\":{\"field\":\"k1\"}}}}",
- EsUtil.toEsDsl(isNullPredicate).toJson());
- Assertions.assertEquals("{\"exists\":{\"field\":\"k1\"}}",
EsUtil.toEsDsl(isNotNullPredicate).toJson());
+ QueryBuilders.toEsDsl(isNullPredicate).toJson());
+ Assertions.assertEquals("{\"exists\":{\"field\":\"k1\"}}",
QueryBuilders.toEsDsl(isNotNullPredicate).toJson());
}
@Test
@@ -228,9 +231,12 @@ public class EsUtilTest extends EsTestCase {
LikePredicate likePredicate1 = new
LikePredicate(LikePredicate.Operator.LIKE, k1, stringLiteral1);
LikePredicate regexPredicate = new
LikePredicate(LikePredicate.Operator.REGEXP, k1, stringLiteral2);
LikePredicate likePredicate2 = new
LikePredicate(LikePredicate.Operator.LIKE, k1, stringLiteral3);
- Assertions.assertEquals("{\"wildcard\":{\"k1\":\"*1*\"}}",
EsUtil.toEsDsl(likePredicate1).toJson());
- Assertions.assertEquals("{\"wildcard\":{\"k1\":\"*1*\"}}",
EsUtil.toEsDsl(regexPredicate).toJson());
- Assertions.assertEquals("{\"wildcard\":{\"k1\":\"1?2\"}}",
EsUtil.toEsDsl(likePredicate2).toJson());
+ Assertions.assertEquals("{\"wildcard\":{\"k1\":\"*1*\"}}",
QueryBuilders.toEsDsl(likePredicate1).toJson());
+ Assertions.assertEquals("{\"wildcard\":{\"k1\":\"*1*\"}}",
QueryBuilders.toEsDsl(regexPredicate).toJson());
+ Assertions.assertEquals("{\"wildcard\":{\"k1\":\"1?2\"}}",
QueryBuilders.toEsDsl(likePredicate2).toJson());
+ List<Expr> notPushDownList = new ArrayList<>();
+ Assertions.assertNull(QueryBuilders.toEsDsl(likePredicate2,
notPushDownList, new HashMap<>(),
BuilderOptions.builder().likePushDown(false).build()));
+ Assertions.assertFalse(notPushDownList.isEmpty());
}
@Test
@@ -243,9 +249,9 @@ public class EsUtilTest extends EsTestCase {
intLiterals.add(intLiteral2);
InPredicate isInPredicate = new InPredicate(k1, intLiterals, false);
InPredicate isNotInPredicate = new InPredicate(k1, intLiterals, true);
- Assertions.assertEquals("{\"terms\":{\"k1\":[3,5]}}",
EsUtil.toEsDsl(isInPredicate).toJson());
+ Assertions.assertEquals("{\"terms\":{\"k1\":[3,5]}}",
QueryBuilders.toEsDsl(isInPredicate).toJson());
Assertions.assertEquals("{\"bool\":{\"must_not\":{\"terms\":{\"k1\":[3,5]}}}}",
- EsUtil.toEsDsl(isNotInPredicate).toJson());
+ QueryBuilders.toEsDsl(isNotInPredicate).toJson());
}
@Test
@@ -257,7 +263,7 @@ public class EsUtilTest extends EsTestCase {
exprs.add(k1);
exprs.add(stringLiteral);
FunctionCallExpr functionCallExpr = new FunctionCallExpr("esquery",
exprs);
- Assertions.assertEquals(str,
EsUtil.toEsDsl(functionCallExpr).toJson());
+ Assertions.assertEquals(str,
QueryBuilders.toEsDsl(functionCallExpr).toJson());
SlotRef k2 = new SlotRef(null, "k2");
IntLiteral intLiteral = new IntLiteral(5);
@@ -266,7 +272,7 @@ public class EsUtilTest extends EsTestCase {
functionCallExpr);
Assertions.assertEquals(
"{\"bool\":{\"must\":[{\"term\":{\"k2\":5}},{\"bool\":{\"must_not\":{\"terms\":{\"k1\":[3,5]}}}}]}}",
- EsUtil.toEsDsl(compoundPredicate).toJson());
+ QueryBuilders.toEsDsl(compoundPredicate).toJson());
}
@Test
@@ -276,7 +282,9 @@ public class EsUtilTest extends EsTestCase {
BinaryPredicate castPredicate = new BinaryPredicate(Operator.EQ,
castExpr, new IntLiteral(3));
List<Expr> notPushDownList = new ArrayList<>();
Map<String, String> fieldsContext = new HashMap<>();
- Assertions.assertNull(EsUtil.toEsDsl(castPredicate, notPushDownList,
fieldsContext));
+ BuilderOptions builderOptions = BuilderOptions.builder()
+
.likePushDown(Boolean.parseBoolean(EsResource.LIKE_PUSH_DOWN_DEFAULT_VALUE)).build();
+ Assertions.assertNull(QueryBuilders.toEsDsl(castPredicate,
notPushDownList, fieldsContext, builderOptions));
Assertions.assertEquals(1, notPushDownList.size());
SlotRef k2 = new SlotRef(null, "k2");
@@ -284,7 +292,8 @@ public class EsUtilTest extends EsTestCase {
BinaryPredicate eqPredicate = new BinaryPredicate(Operator.EQ, k2,
intLiteral);
CompoundPredicate compoundPredicate = new
CompoundPredicate(CompoundPredicate.Operator.OR, castPredicate,
eqPredicate);
- EsUtil.toEsDsl(compoundPredicate, notPushDownList, fieldsContext);
+
+ QueryBuilders.toEsDsl(compoundPredicate, notPushDownList,
fieldsContext, builderOptions);
Assertions.assertEquals(3, notPushDownList.size());
SlotRef k3 = new SlotRef(null, "k3");
@@ -292,7 +301,7 @@ public class EsUtilTest extends EsTestCase {
CastExpr castDoubleExpr = new CastExpr(Type.DOUBLE, k3);
BinaryPredicate castDoublePredicate = new BinaryPredicate(Operator.GE,
castDoubleExpr,
new FloatLiteral(3.0, Type.DOUBLE));
- EsUtil.toEsDsl(castDoublePredicate, notPushDownList, fieldsContext);
+ QueryBuilders.toEsDsl(castDoublePredicate, notPushDownList,
fieldsContext, builderOptions);
Assertions.assertEquals(3, notPushDownList.size());
}
@@ -317,22 +326,20 @@ public class EsUtilTest extends EsTestCase {
+
"\"fields\":{\"keyword\":{\"ignore_above\":256,\"type\":\"keyword\"}}},"
+
"\"test3\":{\"type\":\"double\"},\"test1\":{\"type\":\"keyword\"}}",
testIndex.toJSONString());
- JSONObject testDynamicTemplates = EsUtil.getMappingProps("test",
loadJsonFromFile("data/es/es6_dynamic_templates_mapping.json"),
- "doc");
+ JSONObject testDynamicTemplates = EsUtil.getMappingProps("test",
+
loadJsonFromFile("data/es/es6_dynamic_templates_mapping.json"), "doc");
Assertions.assertEquals("{\"test4\":{\"type\":\"date\"},\"test2\":{\"type\":\"text\","
- +
"\"fields\":{\"keyword\":{\"ignore_above\":256,\"type\":\"keyword\"}}},"
- +
"\"test3\":{\"type\":\"double\"},\"test1\":{\"type\":\"keyword\"}}",
+ +
"\"fields\":{\"keyword\":{\"ignore_above\":256,\"type\":\"keyword\"}}},"
+ +
"\"test3\":{\"type\":\"double\"},\"test1\":{\"type\":\"keyword\"}}",
testDynamicTemplates.toJSONString());
expectedEx.expect(DorisEsException.class);
expectedEx.expectMessage("Do not support index without explicit
mapping.");
- EsUtil.getMappingProps("test",
loadJsonFromFile("data/es/es6_only_dynamic_templates_mapping.json"),
- "doc");
+ EsUtil.getMappingProps("test",
loadJsonFromFile("data/es/es6_only_dynamic_templates_mapping.json"), "doc");
expectedEx.expect(DorisEsException.class);
expectedEx.expectMessage("Do not support index without explicit
mapping.");
- EsUtil.getMappingProps("test",
loadJsonFromFile("data/es/es6_only_dynamic_templates_mapping.json"),
- null);
+ EsUtil.getMappingProps("test",
loadJsonFromFile("data/es/es6_only_dynamic_templates_mapping.json"), null);
}
@Test
@@ -356,17 +363,16 @@ public class EsUtilTest extends EsTestCase {
+
"\"fields\":{\"keyword\":{\"ignore_above\":256,\"type\":\"keyword\"}}},"
+
"\"test3\":{\"type\":\"double\"},\"test1\":{\"type\":\"keyword\"}}",
testIndex.toJSONString());
- JSONObject testDynamicTemplates = EsUtil.getMappingProps("test",
loadJsonFromFile("data/es/es7_dynamic_templates_mapping.json"),
- null);
+ JSONObject testDynamicTemplates = EsUtil.getMappingProps("test",
+
loadJsonFromFile("data/es/es7_dynamic_templates_mapping.json"), null);
Assertions.assertEquals("{\"test4\":{\"type\":\"date\"},\"test2\":{\"type\":\"text\","
- +
"\"fields\":{\"keyword\":{\"ignore_above\":256,\"type\":\"keyword\"}}},"
- +
"\"test3\":{\"type\":\"double\"},\"test1\":{\"type\":\"keyword\"}}",
+ +
"\"fields\":{\"keyword\":{\"ignore_above\":256,\"type\":\"keyword\"}}},"
+ +
"\"test3\":{\"type\":\"double\"},\"test1\":{\"type\":\"keyword\"}}",
testDynamicTemplates.toJSONString());
expectedEx.expect(DorisEsException.class);
expectedEx.expectMessage("Do not support index without explicit
mapping.");
- EsUtil.getMappingProps("test",
loadJsonFromFile("data/es/es7_only_dynamic_templates_mapping.json"),
- null);
+ EsUtil.getMappingProps("test",
loadJsonFromFile("data/es/es7_only_dynamic_templates_mapping.json"), null);
}
@Test
@@ -390,17 +396,16 @@ public class EsUtilTest extends EsTestCase {
+
"\"fields\":{\"keyword\":{\"ignore_above\":256,\"type\":\"keyword\"}}},"
+
"\"test3\":{\"type\":\"double\"},\"test1\":{\"type\":\"keyword\"}}",
testIndex.toJSONString());
- JSONObject testDynamicTemplates = EsUtil.getMappingProps("test",
loadJsonFromFile("data/es/es8_dynamic_templates_mapping.json"),
- "doc");
+ JSONObject testDynamicTemplates = EsUtil.getMappingProps("test",
+
loadJsonFromFile("data/es/es8_dynamic_templates_mapping.json"), "doc");
Assertions.assertEquals("{\"test4\":{\"type\":\"date\"},\"test2\":{\"type\":\"text\","
- +
"\"fields\":{\"keyword\":{\"ignore_above\":256,\"type\":\"keyword\"}}},"
- +
"\"test3\":{\"type\":\"double\"},\"test1\":{\"type\":\"keyword\"}}",
+ +
"\"fields\":{\"keyword\":{\"ignore_above\":256,\"type\":\"keyword\"}}},"
+ +
"\"test3\":{\"type\":\"double\"},\"test1\":{\"type\":\"keyword\"}}",
testDynamicTemplates.toJSONString());
expectedEx.expect(DorisEsException.class);
expectedEx.expectMessage("Do not support index without explicit
mapping.");
- EsUtil.getMappingProps("test",
loadJsonFromFile("data/es/es8_only_dynamic_templates_mapping.json"),
- "doc");
+ EsUtil.getMappingProps("test",
loadJsonFromFile("data/es/es8_only_dynamic_templates_mapping.json"), "doc");
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
