[jira] [Updated] (NIFI-10169) JoinEnrichment merged schema incorrect if first Enrichment record is null
Title: Message Title Pierre Villard updated NIFI-10169 Apache NiFi / NIFI-10169 JoinEnrichment merged schema incorrect if first Enrichment record is null Change By: Pierre Villard Resolution: Fixed Status: Patch Available Resolved Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (NIFI-10169) JoinEnrichment merged schema incorrect if first Enrichment record is null
[
https://issues.apache.org/jira/browse/NIFI-10169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-10169:
--
Fix Version/s: 1.17.0
Status: Patch Available (was: Open)
> JoinEnrichment merged schema incorrect if first Enrichment record is null
> -
>
> Key: NIFI-10169
> URL: https://issues.apache.org/jira/browse/NIFI-10169
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.17.0
>
> Attachments: Enrich.json
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> From the users mailing list:
> {quote}
> I am performing some enrichments, and sometimes the enrichment look up fails
> as the item is not there. If the enrichment fails for the first item in an
> array of records then the merge fails to properly merge subsequent records in
> the same array. The join processor uses an “infer” JSON reader for the
> original leg, a schema based reader for the enriched leg, and a “inherit”
> JSON writer. I am using the “Insert” join strategy. Here is an example.
>
> If the original record is
>
> [{"transport_protocol_id":17,"Enrichment":{}},{"transport_protocol_id":6,"Enrichment":{}}]
>
> Then both lookups succeed, and the enrichment record looks like this
>
> [
> {"network_transport" : {"name" : "udp", "code" : 17, "alias" : "UDP",
> "comment" : "user datagram protocol"}},
> {"network_transport" : {"name" : "tcp", "code" : 6, "alias" : "TCP",
> "comment" : "transmission control protocol"}
> ]
>
> And the joined record looks like this.
>
> [
> {"transport_protocol_id" : 17,"Enrichment" : {
> "network_transport" : {"name" : "udp","code" : 17,"alias" :
> "UDP","comment" : "user datagram protocol"}}},
> {"transport_protocol_id" : 6,"Enrichment" : {
> "network_transport" : {"name" : "tcp","code" : 6,"alias" :
> "TCP","comment" : "transmission control protocol"}}}
> ]
>
> However if the first record has a key value that is out of range, such as this
>
> [{"transport_protocol_id":,"Enrichment":{}},{"transport_protocol_id":6,"Enrichment":{}}]
>
> Then the first record in the enriched leg will be null, even if the rest of
> the records are correct. However the enrichment is still valid JSON once I
> have processed it in the enrichment leg.
>
> [
> {"network_transport" : null},
> {"network_transport" : {"name" : "tcp", "code" : 6,"alias" : "TCP",
> "comment" : "transmission control protocol"}}
> ]
>
> But the joined record does not properly process the subsequent records, and
> the content looks like this.
>
> [
> {"transport_protocol_id" : ,"Enrichment" : {
> "network_transport" : null}},
> {"transport_protocol_id" : 6,"Enrichment" : {
> "network_transport" : "MapRecord[{name=tcp, alias=TCP,
> comment=transmission control protocol, code=6}]"}}
> ]
>
> Is there any step I could use to ensure the join happens as expected? Or is
> this the same situation as the JIRA I mentioned above? I am not able to use a
> schema based writer as our real case has too many input record types and
> enrichment options that the number of combinations, and hence schemas, could
> not be managed.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (NIFI-10169) JoinEnrichment merged schema incorrect if first Enrichment record is null
[
https://issues.apache.org/jira/browse/NIFI-10169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-10169:
--
Attachment: Enrich.json
> JoinEnrichment merged schema incorrect if first Enrichment record is null
> -
>
> Key: NIFI-10169
> URL: https://issues.apache.org/jira/browse/NIFI-10169
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Attachments: Enrich.json
>
>
> From the users mailing list:
> {quote}
> I am performing some enrichments, and sometimes the enrichment look up fails
> as the item is not there. If the enrichment fails for the first item in an
> array of records then the merge fails to properly merge subsequent records in
> the same array. The join processor uses an “infer” JSON reader for the
> original leg, a schema based reader for the enriched leg, and a “inherit”
> JSON writer. I am using the “Insert” join strategy. Here is an example.
>
> If the original record is
>
> [{"transport_protocol_id":17,"Enrichment":{}},{"transport_protocol_id":6,"Enrichment":{}}]
>
> Then both lookups succeed, and the enrichment record looks like this
>
> [
> {"network_transport" : {"name" : "udp", "code" : 17, "alias" : "UDP",
> "comment" : "user datagram protocol"}},
> {"network_transport" : {"name" : "tcp", "code" : 6, "alias" : "TCP",
> "comment" : "transmission control protocol"}
> ]
>
> And the joined record looks like this.
>
> [
> {"transport_protocol_id" : 17,"Enrichment" : {
> "network_transport" : {"name" : "udp","code" : 17,"alias" :
> "UDP","comment" : "user datagram protocol"}}},
> {"transport_protocol_id" : 6,"Enrichment" : {
> "network_transport" : {"name" : "tcp","code" : 6,"alias" :
> "TCP","comment" : "transmission control protocol"}}}
> ]
>
> However if the first record has a key value that is out of range, such as this
>
> [{"transport_protocol_id":,"Enrichment":{}},{"transport_protocol_id":6,"Enrichment":{}}]
>
> Then the first record in the enriched leg will be null, even if the rest of
> the records are correct. However the enrichment is still valid JSON once I
> have processed it in the enrichment leg.
>
> [
> {"network_transport" : null},
> {"network_transport" : {"name" : "tcp", "code" : 6,"alias" : "TCP",
> "comment" : "transmission control protocol"}}
> ]
>
> But the joined record does not properly process the subsequent records, and
> the content looks like this.
>
> [
> {"transport_protocol_id" : ,"Enrichment" : {
> "network_transport" : null}},
> {"transport_protocol_id" : 6,"Enrichment" : {
> "network_transport" : "MapRecord[{name=tcp, alias=TCP,
> comment=transmission control protocol, code=6}]"}}
> ]
>
> Is there any step I could use to ensure the join happens as expected? Or is
> this the same situation as the JIRA I mentioned above? I am not able to use a
> schema based writer as our real case has too many input record types and
> enrichment options that the number of combinations, and hence schemas, could
> not be managed.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (NIFI-10169) JoinEnrichment merged schema incorrect if first Enrichment record is null
[
https://issues.apache.org/jira/browse/NIFI-10169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-10169:
--
Description:
>From the users mailing list:
{quote}
I am performing some enrichments, and sometimes the enrichment look up fails as
the item is not there. If the enrichment fails for the first item in an array
of records then the merge fails to properly merge subsequent records in the
same array. The join processor uses an “infer” JSON reader for the original
leg, a schema based reader for the enriched leg, and a “inherit” JSON writer. I
am using the “Insert” join strategy. Here is an example.
If the original record is
[{"transport_protocol_id":17,"Enrichment":{}},{"transport_protocol_id":6,"Enrichment":{}}]
Then both lookups succeed, and the enrichment record looks like this
[
{"network_transport" : {"name" : "udp", "code" : 17, "alias" : "UDP",
"comment" : "user datagram protocol"}},
{"network_transport" : {"name" : "tcp", "code" : 6, "alias" : "TCP",
"comment" : "transmission control protocol"}
]
And the joined record looks like this.
[
{"transport_protocol_id" : 17,"Enrichment" : {
"network_transport" : {"name" : "udp","code" : 17,"alias" : "UDP","comment"
: "user datagram protocol"}}},
{"transport_protocol_id" : 6,"Enrichment" : {
"network_transport" : {"name" : "tcp","code" : 6,"alias" : "TCP","comment"
: "transmission control protocol"}}}
]
However if the first record has a key value that is out of range, such as this
[{"transport_protocol_id":,"Enrichment":{}},{"transport_protocol_id":6,"Enrichment":{}}]
Then the first record in the enriched leg will be null, even if the rest of the
records are correct. However the enrichment is still valid JSON once I have
processed it in the enrichment leg.
[
{"network_transport" : null},
{"network_transport" : {"name" : "tcp", "code" : 6,"alias" : "TCP", "comment"
: "transmission control protocol"}}
]
But the joined record does not properly process the subsequent records, and the
content looks like this.
[
{"transport_protocol_id" : ,"Enrichment" : {
"network_transport" : null}},
{"transport_protocol_id" : 6,"Enrichment" : {
"network_transport" : "MapRecord[{name=tcp, alias=TCP, comment=transmission
control protocol, code=6}]"}}
]
Is there any step I could use to ensure the join happens as expected? Or is
this the same situation as the JIRA I mentioned above? I am not able to use a
schema based writer as our real case has too many input record types and
enrichment options that the number of combinations, and hence schemas, could
not be managed.
{quote}
was:
>From the users mailing list:
{quote}I am performing some enrichments, and sometimes the enrichment look up
fails as the item is not there. If the enrichment fails for the first item in
an array of records then the merge fails to properly merge subsequent records
in the same array. The join processor uses an “infer” JSON reader for the
original leg, a schema based reader for the enriched leg, and a “inherit” JSON
writer. I am using the “Insert” join strategy. Here is an example.
If the original record is
[\{"transport_protocol_id":17,"Enrichment":{}},\{"transport_protocol_id":6,"Enrichment":{}}]
Then both lookups succeed, and the enrichment record looks like this
[
\{"network_transport" : {"name" : "udp", "code" : 17, "alias" : "UDP",
"comment" : "user datagram protocol"}},
\{"network_transport" : {"name" : "tcp", "code" : 6, "alias" : "TCP",
"comment" : "transmission control protocol"}
]
And the joined record looks like this.
[
{"transport_protocol_id" : 17,"Enrichment" : {
"network_transport" : \{"name" : "udp","code" : 17,"alias" :
"UDP","comment" : "user datagram protocol"}}},
{"transport_protocol_id" : 6,"Enrichment" : {
"network_transport" : \{"name" : "tcp","code" : 6,"alias" : "TCP","comment"
: "transmission control protocol"}}}
]
However if the first record has a key value that is out of range, such as this
[\{"transport_protocol_id":,"Enrichment":{}},\{"transport_protocol_id":6,"Enrichment":{}}]
Then the first record in the enriched leg will be null, even if the rest of the
records are correct. However the enrichment is still valid JSON once I have
processed it in the enrichment leg.
[
\{"network_transport" : null},
\{"network_transport" : {"name" : "tcp", "code" : 6,"alias" : "TCP",
"comment" : "transmission control protocol"}}
]
But the joined record does not properly process the subsequent records, and the
content looks like this.
[
{"transport_protocol_id" : ,"Enrichment" : {
"network_transport" : null}},
{"transport_protocol_id" : 6,"Enrichment" : {
"network_transport" : "MapRecord[\{name=tcp, alias=TCP,
comment=transmission control protocol, code=6}]"}}
]
Is there any step I could use to ensure
