[
https://issues.apache.org/jira/browse/NIFI-16028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095487#comment-18095487
]
ASF subversion and git services commented on NIFI-16028:
--------------------------------------------------------
Commit 7ae29ec40ba106bf79f67fb466f91b85473540fc in nifi's branch
refs/heads/main from agturley
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=7ae29ec40ba ]
NIFI-16028 - Support nested field paths in PutElasticsearchJson (#11355)
Allow the Index Field, Identifier Field, and Timestamp Field properties to
reference a nested field with a "/"-delimited path (e.g. "@metadata/index").
A value with no "/" behaves exactly as before (top-level lookup), so existing
flows are unaffected.
When the corresponding Retain property is false, the extracted leaf is removed
and any ancestor object left empty is pruned recursively, so a transient
envelope like "@metadata" is not indexed once its fields are consumed; parents
that still have other fields are kept.
A field name that literally contains a "/" can be addressed by escaping the
slash as "\/" (and a literal backslash as "\\"); a backslash not followed by
"/" or "\" is left untouched, so existing field names are unaffected. The
escaping rule and worked nesting/escaping examples are documented in the
Index, Identifier, and Timestamp Field property descriptions.
Applies across NDJSON, JSON Array, and Single JSON, and all index operations.
The NDJSON raw-bytes fast path is bypassed only when a nested path is
configured, since the streaming scan only matches flat field names; the path
classification and decoded flat names are resolved once per onTrigger rather
than per record.
Add a Field Path Mode property to PutElasticsearchJson controlling how the
Identifier Field, Index Field, and Timestamp Field values are interpreted:
- Literal Field Name (default): the value is the exact name of a top-level
field, with "/" and "\" treated literally.
- Nested Field Path: the value is a "/"-delimited path into nested objects,
with "\/" and "\\" escaping.
Defaulting to Literal Field Name keeps existing configurations behaving as
before, so no property migration is required. In literal mode the configured
values are escaped into single-segment paths so the shared path helpers
resolve them to the exact field name, leaving the extraction logic unchanged.
Document both modes in additionalDetails.md and add literal-mode tests
alongside the existing nested-path coverage
> PutElasticsearchJson: support nested field paths for Index, Identifier, and
> Timestamp fields
> --------------------------------------------------------------------------------------------
>
> Key: NIFI-16028
> URL: https://issues.apache.org/jira/browse/NIFI-16028
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Reporter: Adam Turley
> Assignee: Adam Turley
> Priority: Minor
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> The PutElasticsearchJson processor can extract the Elasticsearch index name,
> document ID, and @timestamp value from a field within each document (via the
> Index Field, Identifier Field, and Timestamp Field properties added in
> NIFI-15985). Today those properties only match a top-level field name, so a
> document that carries this information in a nested object cannot be used as a
> source. For example, with a document like: \{ "@metadata": { "index":
> "logs-2026", "id": "abc-123" }, "message": "..." } setting Index Field to
> "@metadata/index" currently finds nothing (there is no top-level key
> literally named "@metadata/index") and silently falls back to the configured
> Index property. Nesting routing/metadata under an envelope such as
> "@metadata" is a common pattern (e.g. coming from Logstash), so users have no
> way to get the index/id/timestamp from it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
