github-actions[bot] commented on code in PR #70205:
URL: https://github.com/apache/doris/pull/70205#discussion_r4476111513
##########
fe/fe-connector/fe-connector-es/src/main/java/org/apache/doris/connector/es/EsScanPlanProvider.java:
##########
@@ -299,12 +303,13 @@ private static void
collectSourceUnsafeFields(ConnectorExpression expression, Se
if (expression instanceof ConnectorColumnRef) {
ConnectorColumnRef column = (ConnectorColumnRef) expression;
String type = column.getType().getTypeName();
- // Source arrays preserve their container (or JSON STRING),
whereas ES matches elements
- // and treats an empty array as missing. Source nullness must
retain the local predicate.
+ // Source containers such as {} are nonnull even when ES indexes
no leaf value.
+ // Source arrays also preserve their container (or JSON STRING),
whereas ES matches elements.
// _id comes from hit metadata and cannot acquire an array
representation in source mode.
if (!"_id".equals(column.getColumnName())
&& ("STRING".equalsIgnoreCase(type) ||
"VARCHAR".equalsIgnoreCase(type)
- || "CHAR".equalsIgnoreCase(type) ||
"ARRAY".equalsIgnoreCase(type))) {
+ || "CHAR".equalsIgnoreCase(type) ||
"ARRAY".equalsIgnoreCase(type)
+ || "JSONB".equalsIgnoreCase(type))) {
Review Comment:
[P1] Keep JSONB null predicates local in docvalue mode. The new `JSONB`
branch is only reached when `docValuesMode == 0`, but `resolveDocValuesFields`
currently treats a `flattened` JSONB field as docvalue-capable by default. A
projection containing that field can therefore stay in mode 1 and push `IS
NULL`/`IS NOT NULL` as an ES `exists` query; an empty `{}` source object is
non-null in Doris while it has no indexed flattened leaf. Exclude
JSONB/flattened fields from docvalue mode or mark these predicates unsafe
regardless of scan mode.
##########
be/src/format/table/es/es_scroll_parser.cpp:
##########
@@ -564,36 +583,36 @@ Status get_date_value_int(const rapidjson::Value& col,
PrimitiveType type, bool
template <PrimitiveType T>
Status get_date_int(const rapidjson::Value& col, PrimitiveType type, bool
pure_doc_value,
typename PrimitiveTypeTraits<T>::CppType* slot,
- const cctz::time_zone& time_zone) {
- // this would happend just only when `enable_docvalue_scan = false`, and
field has timestamp format date from _source
+ const cctz::time_zone& time_zone, std::string_view
date_format) {
if (col.IsNumber()) {
- // ES process date/datetime field would use millisecond timestamp for
index or docvalue
- // processing date type field, if a number is encountered, Doris On ES
will force it to be processed according to ms
- // Doris On ES needs to be consistent with ES, so just divided by 1000
because the unit for from_unixtime is seconds
- return get_date_value_int<T>(col, type, false, slot, time_zone);
+ // ES applies the mapped grammar to source numeric tokens too. Numeric
docvalues are
+ // canonical epoch values; requests from an old FE also retain their
millisecond decoder.
+ const bool mapped_source = !pure_doc_value && !date_format.empty();
Review Comment:
[P1] Preserve epoch semantics for source numeric dates. `get_date_int` now
treats every source JSON number as mapped date text whenever `date_format` is
present, so the default `strict_date_optional_time||epoch_millis` interprets
numeric `2024` as the calendar year 2024 instead of the epoch-millis instant
1970-01-01 00:00:02 (and `yyyy-MM-dd` can reject numeric dates entirely).
Elasticsearch numeric date values are always milliseconds since epoch; only
quoted values should use the ordered textual formats. Keep the numeric source
path on the epoch decoder and add a regression for an unquoted 2024.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
