luchunliang opened a new pull request, #12190:
URL: https://github.com/apache/inlong/pull/12190
Fixes #12189
### Motivation
1. **KV map lookup after `STR_TO_MAP`** — decode a URL-encoded query string
and pluck a specific parameter in one line:
```sql
SELECT
STR_TO_MAP(URL_DECODE(event_value), '&', '=')['HY50'] AS HY50
FROM source
WHERE STR_TO_MAP(URL_DECODE(event_value), '&', '=')['HY50'] =
'welfare_milestone_operations'
AND event_code = 'OnPageEnter';
```
2. **JSON key access** — read fields off a raw `JsonObject` produced by JSON
helper functions using the intuitive `obj['field']` syntax, without additional
casting.
3. **Consistency** — align bracket-access semantics with common SQL /
JSONPath expectations, reduce boilerplate, and remove edge cases where a
valid-looking expression silently returned `null`.
### Modifications
`ArrayParser#parse` now handles four shapes; the two new branches (Map,
JsonObject) are highlighted:
```java
@Override
public Object parse(SourceData sourceData, int rowIndex, Context context) {
Object leftValue = this.left.parse(sourceData, rowIndex, context);
Object rightValue = this.right.parse(sourceData, rowIndex, context);
// 1) List indexed by Number (existing)
if (leftValue instanceof List<?> && rightValue instanceof Number) {
return ((List<?>) leftValue).get(((Number) rightValue).intValue());
}
// 2) Map indexed by any key type (NEW)
if (leftValue instanceof Map<?, ?>) {
return ((Map<?, ?>) leftValue).get(rightValue);
}
// 3) JsonArray indexed by Number (existing) — primitives unwrapped
if (leftValue instanceof JsonArray && rightValue instanceof Number) {
JsonElement result = ((JsonArray) leftValue).get(((Number)
rightValue).intValue());
return unwrapJsonElement(result);
}
// 4) JsonObject indexed by String key (NEW) — primitives unwrapped
if (leftValue instanceof JsonObject && rightValue instanceof String) {
JsonElement result = ((JsonObject) leftValue).get((String)
rightValue);
return unwrapJsonElement(result);
}
return null;
}
```
Where `unwrapJsonElement`:
- returns `null` for `JsonNull`;
- returns the natural Java value for `JsonPrimitive` (`String` / `Boolean` /
`Number`, fallback to `toString()`);
- returns the original `JsonArray` / `JsonObject` for structural elements,
so chained bracket access keeps working.
### Behavior matrix
| `leftValue` type | `rightValue` type | Result |
| --- | --- | --- |
| `List<?>` | `Number` | `list.get(index.intValue())` |
| `Map<?, ?>` | any | `map.get(rightValue)` (new) |
| `JsonArray` | `Number` | element at index, primitives unwrapped |
| `JsonObject` | `String` | value at key, primitives unwrapped (new) |
| Any other combination | | `null` |
### Example
Before this change, the following expression silently returned `null` and
the `WHERE` clause could never match:
```sql
SELECT
STR_TO_MAP(URL_DECODE(event_value), '&', '=')['HY50'] AS HY50
FROM source
WHERE STR_TO_MAP(URL_DECODE(event_value), '&', '=')['HY50'] =
'welfare_milestone_operations'
AND event_code = 'OnPageEnter';
```
After this change the map lookup works as intended, so both projection and
filter behave correctly.
### Tests
- Added `TestCsv2RowDataProcessor` (peer of `TestCsv2KvProcessor`) which
drives the CSV → RowData pipeline with a SQL that uses
`STR_TO_MAP(URL_DECODE(event_value), '&', '=')['HY50']` in both `SELECT` and
`WHERE`, exercising the new `Map` branch end-to-end.
- Existing `List` / `JsonArray` bracket-access tests continue to pass
unchanged.
### Backward compatibility
- Fully backward compatible.
- The two existing branches (`List[Number]`, `JsonArray[Number]`) keep
identical semantics.
- The fallback for unrecognized type combinations still returns `null`, so
previously-null expressions can only turn into a real value — never into an
unexpected error.
- No public API / method signature change.
### Risks / Notes
- `Map<?, ?>` uses `map.get(rightValue)` directly; when the SQL parser
resolves the bracket key as a `String` while the map key is a boxed number (or
vice versa), the lookup will return `null` due to `.equals` type mismatch. This
matches Java `Map` semantics and is consistent with how other Transform SDK
operators treat mixed key types.
- For `JsonObject`, missing keys resolve to `null` (Gson returns `null` from
`get`); no exception is thrown.
- `JsonPrimitive` unwrapping intentionally mirrors what `JsonSourceData`
already does elsewhere, so downstream comparison/arithmetic operators receive
the same Java type regardless of whether the value originated from the source
or from a JSON helper function.
### Files changed
-
`inlong-sdk/transform-sdk/src/main/java/org/apache/inlong/sdk/transform/process/parser/ArrayParser.java`
-
`inlong-sdk/transform-sdk/src/test/java/org/apache/inlong/sdk/transform/process/processor/TestCsv2RowDataProcessor.java`
*(new; end-to-end coverage for `Map[key]`)*
### Checklist
- [x] Backward-compatible with existing `List[Number]` / `JsonArray[Number]`
usage
- [x] New unit / integration test added and passing locally
- [x] No changes to public APIs / method signatures
- [x] Behavior of primitive-unwrapping for JSON branches aligned with
`JsonSourceData`
---
### Verifying this change
*(Please pick either of the following options)*
- [ ] This change is a trivial rework/code cleanup without any test coverage.
- [ ] This change is already covered by existing tests, such as:
*(please describe tests)*
- [ ] This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads
(10MB)*
- *Extended integration test for recovery after broker failure*
### Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs
/ not documented)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a follow-up
issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
