Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
markap14 merged PR #7745: URL: https://github.com/apache/nifi/pull/7745 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
markap14 commented on PR #7745: URL: https://github.com/apache/nifi/pull/7745#issuecomment-249151 Thanks for the update @ChrisSamo632 all looks good to me at this point. Sorry this has taken so long, and thanks for sticking with it! +1 will merge to main -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
joewitt commented on PR #7745: URL: https://github.com/apache/nifi/pull/7745#issuecomment-2094286612 Thanks for rebasing @ChrisSamo632 ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on PR #7745: URL: https://github.com/apache/nifi/pull/7745#issuecomment-2094055528 Rebased against latest `main` to refactor changes into new `nifi-extension-bundles` module -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on PR #7745: URL: https://github.com/apache/nifi/pull/7745#issuecomment-1969913218 @markap14 thanks for the review, I've addressed your comments (and reverted the IDE auto-formatting issues!) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
markap14 commented on code in PR #7745: URL: https://github.com/apache/nifi/pull/7745#discussion_r1496517940 ## nifi-docs/src/main/asciidoc/record-path-guide.adoc: ## @@ -945,6 +949,27 @@ The following record path expression would return: | `unescapeJson(/json_str)` | {"name"="John", "age"=30} (as a Map) |== +*Note* that Maps cannot be serialised with a RecordSetWriter. + +Given a record such as: + + +{ + "json_str": "{\"name\":\"John\",\"age\":30,\"addresses\"[{\"address_1\": \"123 Fake Street\"}]}" +} + + +The following record path expression would return: + +|== +| RecordPath | Return value +| `unescapeJson(/json_str, 'false')` | {"name"="John", "age"=30, "addresses"=[{address_1"="123 Fake Street"}]} (as a Map, with each entry in "addresses" as a Map) +| `unescapeJson(/json_str, 'true', 'false')` | {"name": "John", "age": 30, "addresses"=[{address_1"="123 Fake Street"}]} (as a Record, with each entry in "addresses" as a Map) +| `unescapeJson(/json_str, 'true', 'true')` | {"name": "John", "age": 30, "addresses": [{"address_1": "123 Fake Street"}]} (as a Record, with each entry in "addresses" as a Record) +|== + +*Note* that Maps cannot be serialised with a RecordSetWriter. Review Comment: This is the third time that this is stated in the documentation for `unescapeJson`. I think stating it once is enough. ## nifi-docs/src/main/asciidoc/record-path-guide.adoc: ## @@ -894,7 +894,11 @@ The following record path expression would convert the record into an escaped JS === unescapeJson Converts a stringified JSON element to a Record, Array or simple field (e.g. String), using the UTF-8 character set. -Optionally convert JSON Objects parsed as Maps into Records (defaults to false). +Optionally convert JSON Objects parsed as Maps into Records (defaults to false), recursively (defaults to false). Review Comment: Can you clarify this statement? I.e., explicitly state the arguments that the function takes, their names, and whether or not they are required? ## nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestUpdateRecord.java: ## @@ -232,16 +235,20 @@ public void testChangingSchema() throws InitializationException, IOException { final JsonTreeReader jsonReader = new JsonTreeReader(); runner.addControllerService("reader", jsonReader); -final String inputSchemaText = new String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/schema/person-with-name-record.avsc"))); -final String outputSchemaText = new String(Files.readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/schema/person-with-name-string.avsc"))); +final String inputSchemaText = new String(Files + .readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/schema/person-with-name-record.avsc"))); +final String outputSchemaText = new String(Files + .readAllBytes(Paths.get("src/test/resources/TestUpdateRecord/schema/person-with-name-string.avsc"))); -runner.setProperty(jsonReader, SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY, SchemaAccessUtils.SCHEMA_TEXT_PROPERTY); +runner.setProperty(jsonReader, SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY, +SchemaAccessUtils.SCHEMA_TEXT_PROPERTY); runner.setProperty(jsonReader, SchemaAccessUtils.SCHEMA_TEXT, inputSchemaText); runner.enableControllerService(jsonReader); final JsonRecordSetWriter jsonWriter = new JsonRecordSetWriter(); runner.addControllerService("writer", jsonWriter); -runner.setProperty(jsonWriter, SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY, SchemaAccessUtils.SCHEMA_TEXT_PROPERTY); +runner.setProperty(jsonWriter, SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY, +SchemaAccessUtils.SCHEMA_TEXT_PROPERTY); Review Comment: There are a huge number of edits to this file, but almost all of them appear to be inserting arbitrary newlines that are not necessary. Can we fix that? ## nifi-docs/src/main/asciidoc/record-path-guide.adoc: ## @@ -945,6 +949,27 @@ The following record path expression would return: | `unescapeJson(/json_str)` | {"name"="John", "age"=30} (as a Map) |== +*Note* that Maps cannot be serialised with a RecordSetWriter. Review Comment: This seems to be a restatement of the above comment (though the above comment seems more clear to me). Does it make sense to remove this? ## nifi-docs/src/main/asciidoc/record-path-guide.adoc: ## @@ -945,6 +949,27 @@ The following record path expression would return: | `unescapeJson(/json_str)` | {"name"="John", "age"=30} (as a Map) |
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
markap14 commented on PR #7745: URL: https://github.com/apache/nifi/pull/7745#issuecomment-1954700969 Thanks @ChrisSamo632 will take a look this week -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on PR #7745: URL: https://github.com/apache/nifi/pull/7745#issuecomment-1949284657 Have rebased from latest `main` to address merge conflicts, this PR should be ready for re-review @markap14 / @exceptionfactory -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on code in PR #7745: URL: https://github.com/apache/nifi/pull/7745#discussion_r1420401479 ## nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java: ## @@ -469,21 +473,51 @@ public static RecordSchema inferSchema(final Map values, final S final RecordField recordField = new RecordField(key, inferredDataType, true); inferredFieldTypes.add(recordField); -final Object coercedValue = convertType(rawValue, inferredDataType, fieldName, charset); -coercedValues.put(key, coercedValue); +convertType(rawValue, inferredDataType, fieldName, charset); } -final RecordSchema inferredSchema = new SimpleRecordSchema(inferredFieldTypes); -return inferredSchema; +return new SimpleRecordSchema(inferredFieldTypes); } public static Record toRecord(final Object value, final String fieldName, final Charset charset) { +return toRecord(value, fieldName, charset, false); +} + +private static Object covertObjectToRecord(final Object rawValue, final String key, final Charset charset) { +final Object coercedValue; +if (rawValue instanceof Map) { +coercedValue = toRecord(rawValue, key, charset, true); +} else if (rawValue instanceof Object[]) { +final Object[] objArray = (Object[]) rawValue; +coercedValue = Arrays.stream(objArray).noneMatch(o -> o instanceof Map) +? objArray +: Arrays.stream(objArray).map(o -> toRecord(o, key, charset, true)).toArray(); +} else if (rawValue instanceof Collection) { +final Collection objCollection = (Collection) rawValue; +// Records have ARRAY DataTypes, so convert any Collections +coercedValue = objCollection.stream().noneMatch(o -> o instanceof Map) +? objCollection.toArray() +: objCollection.stream().map(o -> toRecord(o, key, charset, true)).toArray(); Review Comment: That's fair, I've reverted this and found a couple of other Streams within `DataTypeUtils` that should probably be changed too, will include in this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on code in PR #7745: URL: https://github.com/apache/nifi/pull/7745#discussion_r1420713574 ## nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java: ## @@ -1837,7 +1838,41 @@ public void testUnescapeJson() { put("firstName", "John"); put("age", 30); }}, "json_str"), -RecordPath.compile("unescapeJson(/json_str, 'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue() +RecordPath.compile("unescapeJson(/json_str, 'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(AssertionError::new).getValue() +); + +// test nested Record converted from Map Object +final Record nestedRecordFromMap = new MapRecord(schema, +Collections.singletonMap( +"json_str", + "{\"firstName\":\"John\",\"age\":30,\"addresses\":[{\"address_1\":\"123 Fake Street\"}]}") +); +// recursively convert Maps to Records (addresses becomes and ARRAY or RECORDs) +assertEquals( +DataTypeUtils.toRecord(new LinkedHashMap(){{ +put("firstName", "John"); +put("age", 30); +put("addresses", new Object[] {DataTypeUtils.toRecord(Collections.singletonMap("address_1", "123 Fake Street"), "addresses")}); +}}, "json_str"), Review Comment: Absolutely. The only problem is that `Map.of` doesn't provide deterministic ordering, so we do need to use `LinkedHashMap`s in some places, which I've changed to use the approach suggested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on PR #7745: URL: https://github.com/apache/nifi/pull/7745#issuecomment-1878261969 @markap14 grateful if you've time to re-review this change at some point -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on code in PR #7745: URL: https://github.com/apache/nifi/pull/7745#discussion_r1420713574 ## nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java: ## @@ -1837,7 +1838,41 @@ public void testUnescapeJson() { put("firstName", "John"); put("age", 30); }}, "json_str"), -RecordPath.compile("unescapeJson(/json_str, 'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue() +RecordPath.compile("unescapeJson(/json_str, 'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(AssertionError::new).getValue() +); + +// test nested Record converted from Map Object +final Record nestedRecordFromMap = new MapRecord(schema, +Collections.singletonMap( +"json_str", + "{\"firstName\":\"John\",\"age\":30,\"addresses\":[{\"address_1\":\"123 Fake Street\"}]}") +); +// recursively convert Maps to Records (addresses becomes and ARRAY or RECORDs) +assertEquals( +DataTypeUtils.toRecord(new LinkedHashMap(){{ +put("firstName", "John"); +put("age", 30); +put("addresses", new Object[] {DataTypeUtils.toRecord(Collections.singletonMap("address_1", "123 Fake Street"), "addresses")}); +}}, "json_str"), Review Comment: Absolutely. The only problem is that `Map.of` doesn't provide deterministic ordering, so we do need to use `LinkedHasMap`s in some places, which I've changed to use the approach suggested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
exceptionfactory commented on code in PR #7745: URL: https://github.com/apache/nifi/pull/7745#discussion_r1420465749 ## nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java: ## @@ -1837,7 +1838,41 @@ public void testUnescapeJson() { put("firstName", "John"); put("age", 30); }}, "json_str"), -RecordPath.compile("unescapeJson(/json_str, 'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue() +RecordPath.compile("unescapeJson(/json_str, 'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(AssertionError::new).getValue() +); + +// test nested Record converted from Map Object +final Record nestedRecordFromMap = new MapRecord(schema, +Collections.singletonMap( +"json_str", + "{\"firstName\":\"John\",\"age\":30,\"addresses\":[{\"address_1\":\"123 Fake Street\"}]}") +); +// recursively convert Maps to Records (addresses becomes and ARRAY or RECORDs) +assertEquals( +DataTypeUtils.toRecord(new LinkedHashMap(){{ +put("firstName", "John"); +put("age", 30); +put("addresses", new Object[] {DataTypeUtils.toRecord(Collections.singletonMap("address_1", "123 Fake Street"), "addresses")}); +}}, "json_str"), Review Comment: Yes, Map.of() is much better in many ways, and it is unfortunate that some portions of test code use the initialization approach out of convenience. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on code in PR #7745: URL: https://github.com/apache/nifi/pull/7745#discussion_r1420434506 ## nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java: ## @@ -1837,7 +1838,41 @@ public void testUnescapeJson() { put("firstName", "John"); put("age", 30); }}, "json_str"), -RecordPath.compile("unescapeJson(/json_str, 'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue() +RecordPath.compile("unescapeJson(/json_str, 'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(AssertionError::new).getValue() +); + +// test nested Record converted from Map Object +final Record nestedRecordFromMap = new MapRecord(schema, +Collections.singletonMap( +"json_str", + "{\"firstName\":\"John\",\"age\":30,\"addresses\":[{\"address_1\":\"123 Fake Street\"}]}") +); +// recursively convert Maps to Records (addresses becomes and ARRAY or RECORDs) +assertEquals( +DataTypeUtils.toRecord(new LinkedHashMap(){{ +put("firstName", "John"); +put("age", 30); +put("addresses", new Object[] {DataTypeUtils.toRecord(Collections.singletonMap("address_1", "123 Fake Street"), "addresses")}); +}}, "json_str"), Review Comment: I've never been a particular fan of the `{{ ... }}` constructor approach, but I've seen it throughout lots of NiFi modules, so had just been copying the style. With the advent of `Map.of`, that's a better approach anyway I think -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on code in PR #7745: URL: https://github.com/apache/nifi/pull/7745#discussion_r1420402511 ## nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java: ## @@ -1837,7 +1838,41 @@ public void testUnescapeJson() { put("firstName", "John"); put("age", 30); }}, "json_str"), -RecordPath.compile("unescapeJson(/json_str, 'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue() +RecordPath.compile("unescapeJson(/json_str, 'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(AssertionError::new).getValue() +); + +// test nested Record converted from Map Object +final Record nestedRecordFromMap = new MapRecord(schema, +Collections.singletonMap( +"json_str", + "{\"firstName\":\"John\",\"age\":30,\"addresses\":[{\"address_1\":\"123 Fake Street\"}]}") Review Comment: I'd been sticking with Java 8 compatible code to get this onto the `support/1.x` branch, but if we're happy with it being only NiFi 2.x now, I'll update these tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
ChrisSamo632 commented on code in PR #7745: URL: https://github.com/apache/nifi/pull/7745#discussion_r1420401479 ## nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java: ## @@ -469,21 +473,51 @@ public static RecordSchema inferSchema(final Map values, final S final RecordField recordField = new RecordField(key, inferredDataType, true); inferredFieldTypes.add(recordField); -final Object coercedValue = convertType(rawValue, inferredDataType, fieldName, charset); -coercedValues.put(key, coercedValue); +convertType(rawValue, inferredDataType, fieldName, charset); } -final RecordSchema inferredSchema = new SimpleRecordSchema(inferredFieldTypes); -return inferredSchema; +return new SimpleRecordSchema(inferredFieldTypes); } public static Record toRecord(final Object value, final String fieldName, final Charset charset) { +return toRecord(value, fieldName, charset, false); +} + +private static Object covertObjectToRecord(final Object rawValue, final String key, final Charset charset) { +final Object coercedValue; +if (rawValue instanceof Map) { +coercedValue = toRecord(rawValue, key, charset, true); +} else if (rawValue instanceof Object[]) { +final Object[] objArray = (Object[]) rawValue; +coercedValue = Arrays.stream(objArray).noneMatch(o -> o instanceof Map) +? objArray +: Arrays.stream(objArray).map(o -> toRecord(o, key, charset, true)).toArray(); +} else if (rawValue instanceof Collection) { +final Collection objCollection = (Collection) rawValue; +// Records have ARRAY DataTypes, so convert any Collections +coercedValue = objCollection.stream().noneMatch(o -> o instanceof Map) +? objCollection.toArray() +: objCollection.stream().map(o -> toRecord(o, key, charset, true)).toArray(); Review Comment: That's fair, I've reverted this and found a coupleof other Streams within `DataTypeUtils` that should probably be changed too, will include in this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-8134 allow unescapeJson Record Path function to recursively convert Maps to Records [nifi]
markap14 commented on code in PR #7745: URL: https://github.com/apache/nifi/pull/7745#discussion_r1417519143 ## nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java: ## @@ -1837,7 +1838,41 @@ public void testUnescapeJson() { put("firstName", "John"); put("age", 30); }}, "json_str"), -RecordPath.compile("unescapeJson(/json_str, 'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue() +RecordPath.compile("unescapeJson(/json_str, 'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(AssertionError::new).getValue() +); + +// test nested Record converted from Map Object +final Record nestedRecordFromMap = new MapRecord(schema, +Collections.singletonMap( +"json_str", + "{\"firstName\":\"John\",\"age\":30,\"addresses\":[{\"address_1\":\"123 Fake Street\"}]}") +); +// recursively convert Maps to Records (addresses becomes and ARRAY or RECORDs) +assertEquals( +DataTypeUtils.toRecord(new LinkedHashMap(){{ +put("firstName", "John"); +put("age", 30); +put("addresses", new Object[] {DataTypeUtils.toRecord(Collections.singletonMap("address_1", "123 Fake Street"), "addresses")}); +}}, "json_str"), Review Comment: We should be creating subclasses of LinkedHashMap for initialization purposes. Should just declare a LinkedHashMap and add elements to it: ``` Map values = new LinkedHashMap<>(); values.put("firstName", "John"); values.put("age", 30); values.put("addresses", new Object[] {DataTypeUtils.toRecord(Collections.singletonMap("address_1", "123 Fake Street"), "addresses")}); ``` ## nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java: ## @@ -1837,7 +1838,41 @@ public void testUnescapeJson() { put("firstName", "John"); put("age", 30); }}, "json_str"), -RecordPath.compile("unescapeJson(/json_str, 'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue() +RecordPath.compile("unescapeJson(/json_str, 'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(AssertionError::new).getValue() +); + +// test nested Record converted from Map Object +final Record nestedRecordFromMap = new MapRecord(schema, +Collections.singletonMap( +"json_str", + "{\"firstName\":\"John\",\"age\":30,\"addresses\":[{\"address_1\":\"123 Fake Street\"}]}") Review Comment: Would probably be cleaner here to use a text block here: ``` """ {"firstName":"John","age":30,"address"...}""" ``` ## nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java: ## @@ -469,21 +473,51 @@ public static RecordSchema inferSchema(final Map values, final S final RecordField recordField = new RecordField(key, inferredDataType, true); inferredFieldTypes.add(recordField); -final Object coercedValue = convertType(rawValue, inferredDataType, fieldName, charset); -coercedValues.put(key, coercedValue); +convertType(rawValue, inferredDataType, fieldName, charset); } -final RecordSchema inferredSchema = new SimpleRecordSchema(inferredFieldTypes); -return inferredSchema; +return new SimpleRecordSchema(inferredFieldTypes); } public static Record toRecord(final Object value, final String fieldName, final Charset charset) { +return toRecord(value, fieldName, charset, false); +} + +private static Object covertObjectToRecord(final Object rawValue, final String key, final Charset charset) { Review Comment: typo - name is `covert...` instead of `convert...` But we should not have a method named `convertObjectToRecord` that returns an Object that may be an array, may be a Record, or may be any other object that was provided. Need to either always return a Record or create a method name that reflects what is truly done. ## nifi-commons/nifi-record-path/src/test/java/org/apache/nifi/record/path/TestRecordPath.java: ## @@ -1837,7 +1838,41 @@ public void testUnescapeJson() { put("firstName", "John"); put("age", 30); }}, "json_str"), -RecordPath.compile("unescapeJson(/json_str, 'true')").evaluate(recordFromMap).getSelectedFields().findFirst().orElseThrow(IllegalStateException::new).getValue() +Record