luchunliang opened a new pull request, #12184:
URL: https://github.com/apache/inlong/pull/12184
Fixes #12183
### Description
When users configure field-mapping SQL against a JSON / Protobuf source in
Transform SDK, two developer-experience gaps exist today:
1. **No way to obtain the row index** inside a child-array (multi-row)
source.
When a source is configured with `childRoot` (e.g. `msgs` array), the
decoder emits multiple rows. Users often need a monotonically-increasing index
per output row (for de-duplication keys, ordered writes, debug, etc.). There is
no built-in placeholder to reference this index in the `SELECT` expression.
2. **`$root.` prefix is mandatory but verbose.**
To reference a root-level JSON field `session_id`, users must write
`$root.session_id`. This is redundant for the very common case where the field
is on the root object. Users coming from typical SQL / JSONPath expect to just
write `session_id` or `person.name`.
This issue proposes to:
- Introduce a new reserved placeholder **`$childIndex`** that resolves to
the current row index (0-based).
- Allow **omitting the `$root` prefix** for JSON field references.
`session_id` behaves identically to `$root.session_id`; `person.address.city`
behaves identically to `$root.person.address.city`. The two styles can be mixed
in the same SQL.
- Keep the existing `$root.*` and `$child.*` semantics **fully backward
compatible** — no behavior change for existing configs.
### Motivation / Use Cases
- **`$childIndex`**
- Add a stable ordering column when flattening a JSON array into multiple
rows.
- Build composite deduplication keys `(sid, $childIndex)` for downstream
sinks that require primary keys.
- Debugging / observability of per-element position without adding it
upstream.
- **Omitted `$root` prefix**
- Reduce boilerplate in mapping SQL; align with common JSONPath / SQL
intuition.
- Simplify auto-generated mapping expressions in the manager / UI.
### Proposed Change
#### 1) `JsonSourceData`
- Add public constant `CHILD_INDEX_KEY = "$childIndex"`.
- In `getField(int rowNum, String fieldName)`, short-circuit when
`fieldName` equals `$childIndex` and return `rowNum`. This works for both
single-row sources (returns `0`) and multi-row child-array sources (returns the
child index).
- In `getFieldByElement`, when the first path segment is neither `$root` nor
`$child`, treat the reference as rooted at `root` and virtually prepend `$root`
to the parsed node list. This makes `person.name` equivalent to
`$root.person.name`.
Key snippets (already implemented):
```java
// JsonSourceData#getField
if (StringUtils.equals(CHILD_INDEX_KEY, fieldName)) {
return rowNum;
}
```
```java
// JsonSourceData#getFieldByElement
} else {
// default root node
current = root;
childNodes.add(0, new JsonNode(ROOT_KEY));
}
```
#### 2) `PbSourceData` (aligned semantics for Protobuf source)
- Add public constant `CHILD_INDEX_KEY = "$childIndex"`.
- In `getField(int rowNum, String fieldName)`, short-circuit when
`fieldName` equals `$childIndex` and return `rowNum`.
- `findFieldNode` already falls back to parsing under `rootDesc` when the
field name has no `$root.` / `$child.` prefix, giving the PB source the same
omitted-prefix behavior as JSON.
Key snippet:
```java
// PbSourceData#getField
if (StringUtils.equals(CHILD_INDEX_KEY, fieldName)) {
return rowNum;
}
```
#### 3) Tests
Add 6 new cases in `TestJson2RowDataProcessor` covering the two features and
their combination:
| # | Test | Feature
| Assertion
|
| --- | --------------------------------------------- |
-------------------------------------- |
---------------------------------------------------------------- |
| 1 | `testJsonChildIndexMapping` | `$childIndex` on
multi-row child array | 3 rows get indices `0` / `1` / `2`
|
| 2 | `testJsonChildIndexSingleRow` | `$childIndex` on
single-row source | Index is `0`
|
| 3 | `testJsonOmitRootPrefix` | Omit `$root` on flat
fields | `session_id` ≡ `$root.session_id`
|
| 4 | `testJsonOmitRootPrefixWithNestedPath` | Omit `$root` on
nested path | `person.address.city` ≡ `$root.person.address.city`
|
| 5 | `testJsonOmitRootPrefixMixedWithExplicitRoot` | Mix of both styles
| Explicit and omitted forms produce identical results
|
| 6 | `testJsonChildIndexWithOmittedRootPrefix` | Combined scenario
| `$childIndex` + omitted `$root` + `$child.*` coexist
|
### Example
**Before** (verbose):
```sql
SELECT
$root.session_id AS session_id,
$root.business AS business,
$child.msg AS msg
FROM source;
```
**After** (equivalent, with the new features):
```sql
SELECT
$childIndex AS row_idx, -- NEW: 0-based row index
session_id AS session_id, -- NEW: $root prefix omitted
business AS business, -- NEW: $root prefix omitted
$child.msg AS msg
FROM source;
```
### Backward Compatibility
- Fully backward compatible.
- `$root.*` and `$child.*` continue to work exactly as before.
- `$childIndex` is a new reserved token; users who did not previously have a
field literally named `$childIndex` are not affected.
- No public API signature change; only new constants plus one new
short-circuit branch in `getField`, and a fallback branch in
`getFieldByElement`.
### Risks / Notes
- A field literally named `$childIndex` in the source payload will no longer
be reachable via `$childIndex` — it must be referenced as `$root.$childIndex`.
Given that `$`-prefixed field names are extremely uncommon in JSON / PB
payloads, this is considered acceptable.
- The omitted-prefix fallback is only applied when the first path segment is
not `$root` or `$child`, so it cannot shadow existing behavior.
### Files Changed
-
`inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/decode/JsonSourceData.java`
-
`inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/decode/PbSourceData.java`
-
`inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/processor/TestJson2RowDataProcessor.java`
### Checklist
- [x] Backward-compatible with existing `$root.*` / `$child.*` field mappings
- [x] New unit tests added and passing locally
- [x] No changes to public APIs / method signatures
- [x] Behavior aligned between JSON source and PB source
---
### Verifying this change
*(Please pick either of the following options)*
- [ ] This change is a trivial rework/code cleanup without any test coverage.
- [ ] This change is already covered by existing tests, such as:
*(please describe tests)*
- [ ] This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads
(10MB)*
- *Extended integration test for recovery after broker failure*
### Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs
/ not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a follow-up
issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
