Hello Bjørn,
If the size of the encoded XML document is small (under ~1 KB), you can extract
the Base64-encoded value to a flowfile attribute using EvaluateJSONPath,
perform the decoding using the base64Decode Expression Language function [1],
and then replace it into the flowfile JSON content using ReplaceText (using
some regex like "content": ".*" -> “content": ”${decodedXML}” where decodedXML
is the name of the attribute you are using).
If the XML content could be very large, this will negatively affect your
performance, as attributes are stored directly in memory and handling large
amounts of data will impact the heap. In this case, I would recommend writing a
Groovy script in ExecuteScript processor to leverage Groovy’s very friendly
JSON handling and extract the value, Base64 decode it, and replace it in a
couple lines.
Hope this helps.
[1]
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#base64decode
<https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#base64decode>
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 24, 2020, at 4:24 AM, Myklebust, Bjørn Magnar
> <bjorn.mykleb...@skatteetaten.no> wrote:
>
>
> Hi.
> I have a set of Json-files which contain a base64-coded field (Jsonpath to
> this field is $.data.content), and this field contains a XML-document.
> Decoding the field works as expected, so does the conversion from xml to
> json, and I'm able to write the content from this field to a file in a
> bucket in S3. But what I would like to do is to be able to replace the coded
> value for this field in the original file with the decoded/converted value in
> stead of writing the decoded/converted value to file. And after replacing the
> json-value then I can write the updated Json-file to a new S3 bucket.
> My process look like this at the moment, and works fine for getting the data
> to file, but it's missing the last part of replacing $.data.content with the
> decoded/converted data.
>
> So how can I do the last part?
>
> <pastedImage.png>
>
> The EvaluedJsonPath looks like this:
>
> <EvaluateJsonPath.png>
>
>
> The ReplaceText looks like this:
>
> <ReplaceText.png>
>
> The Base64EncodeContent looks like this:
>
> <Base64Decode.png>
>
> and finally, the CovertRecord looks like this:
>
> <ConvertRecord.png>
>
>
>
> This is a testfile for that I'm working with:
>
> {
> "header": {
> "dokumentidentifikator": null,
> "dokumentidentifikatorV2": "dcff985b-c652-4085-b8f1-45a2f4b6d150",
> "revisjonsnummer": 1,
> "dokumentnavn":
> "Engangsavgiftfastsettelse:55TEST661122334455:44BIL1:2017-10-20",
> "dokumenttype": "SKATTEMELDING_ENGANGSAVGIFT",
> "dokumenttilstand": "OPPRETTET",
> "gyldig": true,
> "gjelderInntektsaar": 2017,
> "gjelderPeriode": "2017_10",
> "gjelderPart": {
> "partsnummer": 5544332211,
> "identifiseringstype": "MASKINELL",
> "identifikator": null
> },
> "opphavspart": {
> "partsnummer": 5544332211,
> "identifikator": null
> },
> "kildereferanse": {
> "kildesystem": "ENGANGSAVGIFTFASTSETTELSE",
> "gruppe": "",
> "referanse": "aef147fb-8ce8-43ef-833b-7aa3bac1ece0",
> "tidspunkt": "2018-01-16T13:28:02.49+01:00"
> }
> },
> "data": {
> "metadata": {
> "format": "ske:fastsetting:motorvogn:motorvognavgift:v1",
> "bytes": 4420,
> "mimeType": "application/xml",
> "sha1": "c0AowOsTdNdo6VufeSsZqTphc0Y="
> },
> "content":
> "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9InllcyI/Pgo8bW90b3J2b2duYXZnaWZ0IHhtbG5zPSJza2U6ZmFzdHNldHRpbmc6bW90b3J2b2duOm1vdG9ydm9nbmF2Z2lmdDp2MSI+CiAgICA8YXZnaWZ0c2xpbmplPgogICAgICAgIDxhdmdpZnRzYmVsb2VwPjU0Mjg5Ni4wMDwvYXZnaWZ0c2JlbG9lcD4KICAgICAgICA8YXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgICAgIDxzYWVyYXZnaWZ0VHlwZWtvZGU+QkI8L3NhZXJhdmdpZnRUeXBla29kZT4KICAgICAgICAgICAgPHNhZXJhdmdpZnRHcnVwcGVrb2RlPlg8L3NhZXJhdmdpZnRHcnVwcGVrb2RlPgogICAgICAgIDwvYXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgPGF2Z2lmdHNkYXRvPjIwMTctMTAtMjA8L2F2Z2lmdHNkYXRvPgogICAgPC9hdmdpZnRzbGluamU+CiAgICA8YmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICAgICAgPGtpZG51bW1lcj4xMDEwMTAxMDEwMTA8L2tpZG51bW1lcj4KICAgICAgICA8Zm9yZmFsbHNkYXRvPjIwMTctMTAtMjA8L2ZvcmZhbGxzZGF0bz4KICAgICAgICA8ZmFrdHVyYWRhdG8+MjAxNy0xMC0yMDwvZmFrdHVyYWRhdG8+CiAgICAgICAgPHRvdGFsQXZnaWZ0c2JlbG9lcD41NDI4OTYuMDA8L3RvdGFsQXZnaWZ0c2JlbG9lcD4KICAgIDwvYmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICA8bW90b3J2b2duYXZnaWZ0c3R5cGU+ZW5nYW5nc2F2Z2lmdDwvbW90b3J2b2duYXZnaWZ0c3R5cGU+CiAgICA8dGlkc3N0ZW1wZWw+MjAxOC0wMS0xNiswMTowMDwvdGlkc3N0ZW1wZWw+CiAgICA8Z3J1bm5sYWdGb3JNb3RvcnZvZ25hdmdpZnQ+CiAgICAgICAgPGtqb2VyaW5nZW5zQXJ0PjEwPC9ram9lcmluZ2Vuc0FydD4KICAgICAgICA8a2pvZXJldG9leT4KICAgICAgICAgICAgPGVpZXJza2FwUmVnaXN0cmVydD4yMDE3LTEwLTIwPC9laWVyc2thcFJlZ2lzdHJlcnQ+CiAgICAgICAgICAgIDxmb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4yMDE3PC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4KICAgICAgICAgICAgPGZvZXJzdGVSZWdpc3RyZXJpbmdzZGF0b0lOb3JnZT4yMDE3LTEwLTIwPC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2RhdG9JTm9yZ2U+CiAgICAgICAgICAgIDxram9lcmV0b2V5Z3J1cHBlPjEwMTwva2pvZXJldG9leWdydXBwZT4KICAgICAgICAgICAgPGxlbmdkZT4zOTY0PC9sZW5nZGU+CiAgICAgICAgICAgIDxtb3RvcmVmZmVrdD45NjwvbW90b3JlZmZla3Q+CiAgICAgICAgICAgIDxzbGFndm9sdW0+MTQzPC9zbGFndm9sdW0+CiAgICAgICAgICAgIDxkcml2c3RvZmY+QkVOU0lOPC9kcml2c3RvZmY+CiAgICAgICAgICAgIDxlZ2VudmVrdD4xNTE5PC9lZ2VudmVrdD4KICAgICAgICAgICAgPGVpZXI+CiAgICAgICAgICAgICAgICA8Zm9lZHNlbHNFbGxlckRudW1tZXI+MTEyMjMzNDQ1NTwvZm9lZHNlbHNFbGxlckRudW1tZXI+CiAgICAgICAgICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICAgICAgICAgICAgICA8bmF2bj5LTEFSQSBLVTwvbmF2bj4KICAgICAgICAgICAgPC9laWVyPgogICAgICAgICAgICA8dGlsbGF0dFRvdGFsdmVrdD4yMTY0PC90aWxsYXR0VG90YWx2ZWt0PgogICAgICAgICAgICA8aHlicmlkPm5laTwvaHlicmlkPgogICAgICAgICAgICA8Y28ydXRzbGlwcD4yNjg8L2NvMnV0c2xpcHA+CiAgICAgICAgICAgIDxub3h1dHNsaXBwPjU5LjQ8L25veHV0c2xpcHA+CiAgICAgICAgICAgIDxram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICAgICAgICAgIDxram9lcmV0b2V5VW5pa0lkZW50aWZpa2F0b3I+QUJDREVGR0hJSjwva2pvZXJldG9leVVuaWtJZGVudGlmaWthdG9yPgogICAgICAgICAgICAgICAgPGtqZW5uZW1lcmtlPjQ0QklMMTwva2plbm5lbWVya2U+CiAgICAgICAgICAgICAgICA8dW5kZXJzdGVsbHNudW1tZXI+VU5ERVJTVEVMTDQ0PC91bmRlcnN0ZWxsc251bW1lcj4KICAgICAgICAgICAgPC9ram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICA8L2tqb2VyZXRvZXk+CiAgICA8L2dydW5ubGFnRm9yTW90b3J2b2duYXZnaWZ0PgogICAgPGF2Z2lmdHNwbGlrdGlnPgogICAgICAgIDxmb2Vkc2Vsc0VsbGVyRG51bW1lcj4xMTIyMzM0NDU1PC9mb2Vkc2Vsc0VsbGVyRG51bW1lcj4KICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICA8L2F2Z2lmdHNwbGlrdGlnPgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzI8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjQ3NTMzNy4yMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5FZ2VudmVrdDwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NjA5NDUuNjQ8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3Q8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40MjEzLjI0PC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IEZyYXRyZWtrPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bSBGcmF0cmVrazwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40NzUzMzcuMjA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD42MDk0NS42NDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5Nb3RvcmVmZmVrdCBTdW08L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IFN1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NDIxMy4yNDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5TbGFndm9sdW0gU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlZyYWtwYW50PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4yNDAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PkJydWtzZnJhZHJhZyAwJTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGZvcmhhYW5kc2JlcmVnbmluZz5mYWxzZTwvZm9yaGFhbmRzYmVyZWduaW5nPgo8L21vdG9ydm9nbmF2Z2lmdD4K"
> },
> "extension": null,
> "skjemaversjon": "v3_0"
> }
>
>
>
>
> Cheers,
> Bjørn
>
>
> Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den
> institusjon eller person den er rettet til og kan være belagt med lovbestemt
> taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt
> Skatteetaten.
> The contents of this email message and any attachments are intended solely
> for the addressee(s) and may contain confidential information and may be
> legally protected from disclosure. If you are not the intended recipient of
> this message, please immediately delete the message and alert the Norwegian
> Tax Administration.
