Thanks, Andy. No, I’m sure there is no reason for that – it’s just that I’m fairly new to NiFi and don’t know it too well yet.
Thanks, Bjørn Fra: Andy LoPresto <alopre...@apache.org> Sendt: tirsdag 30. juni 2020 18:37 Til: users@nifi.apache.org Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value You should not need to explicitly set the additional module directory to cover that location. Is there a reason you can’t use the native Groovy JSON [1] parsing? That way you don’t have to download any additional libraries. [1] http://groovy-lang.org/json.html#<http://groovy-lang.org/json.html> Andy LoPresto alopre...@apache.org<mailto:alopre...@apache.org> alopresto.apa...@gmail.com<mailto:alopresto.apa...@gmail.com> He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 On Jun 29, 2020, at 7:41 AM, Myklebust, Bjørn Magnar <bjorn.mykleb...@skatteetaten.no<mailto:bjorn.mykleb...@skatteetaten.no>> wrote: Andy, just a quick followup on this. I wanted to test a groovy-script with this code (not finished by far yet), and the script is placed in the Script Body part of an ExecuteGroovyScript-process in NiFi: import org.json.JSONObject import org.json.XML import org.apache.commons.io.IOUtils import java.nio.charset.* def flowFile = session.get() if (!flowFile) return flowFile = session.write(flowFile, {inputStream, outputStream -> def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8) def xmlJSONObj = XML.toJSONObject(text); def json = xmlJSONObj.toString(); outputStream.write(json.getBytes(StandardCharsets.UTF_8)) } as StreamCallback) session.transfer(flowFile, ExecuteScript.REL_SUCCESS) But when trying to run this I get the message «unable to resolve class org.json.JSONObject @ line 1» I have downloaded the jar file from this site: https://repo1.maven.org/maven2/org/json/json/20200518/json-20200518.jar And placed it in my nifi/lib-directory. And the content of this jar you can see in the enclosed png-picture. Do I need to set a value for the property Additional Classpath when the jar-file is stored in the lib-directory? Thanks, Bjørn Fra: Andy LoPresto <alopre...@apache.org<mailto:alopre...@apache.org>> Sendt: torsdag 25. juni 2020 19:20 Til: users@nifi.apache.org<mailto:users@nifi.apache.org> Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value Hi Bjørn, No, XML to JSON conversion is not an Expression Language feature. You’ll need to either get this data into a flowfile as the complete content to perform the conversion with existing built-in tools, or add that step to your Groovy script. With that additional requirement, I think using the Groovy script to perform those steps in tandem is probably the most performant and logical approach here. Andy LoPresto alopre...@apache.org<mailto:alopre...@apache.org> alopresto.apa...@gmail.com<mailto:alopresto.apa...@gmail.com> He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 On Jun 24, 2020, at 11:25 PM, Myklebust, Bjørn Magnar <bjorn.mykleb...@skatteetaten.no<mailto:bjorn.mykleb...@skatteetaten.no>> wrote: Thanks Andy. The XML-content is around 5 kB-ish. But I also need to convert the XML to JSON before replacing it back into the original JSON-file. Can this be done with e.g a ConvertAttribute before the ReplaceText? Thanks, Bjørn Fra: Andy LoPresto <alopre...@apache.org<mailto:alopre...@apache.org>> Sendt: onsdag 24. juni 2020 17:24 Til: users@nifi.apache.org<mailto:users@nifi.apache.org> Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value Hello Bjørn, If the size of the encoded XML document is small (under ~1 KB), you can extract the Base64-encoded value to a flowfile attribute using EvaluateJSONPath, perform the decoding using the base64Decode Expression Language function [1], and then replace it into the flowfile JSON content using ReplaceText (using some regex like "content": ".*" -> “content": ”${decodedXML}” where decodedXML is the name of the attribute you are using). If the XML content could be very large, this will negatively affect your performance, as attributes are stored directly in memory and handling large amounts of data will impact the heap. In this case, I would recommend writing a Groovy script in ExecuteScript processor to leverage Groovy’s very friendly JSON handling and extract the value, Base64 decode it, and replace it in a couple lines. Hope this helps. [1] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#base64decode Andy LoPresto alopre...@apache.org<mailto:alopre...@apache.org> alopresto.apa...@gmail.com<mailto:alopresto.apa...@gmail.com> He/Him PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 On Jun 24, 2020, at 4:24 AM, Myklebust, Bjørn Magnar <bjorn.mykleb...@skatteetaten.no<mailto:bjorn.mykleb...@skatteetaten.no>> wrote: Hi. I have a set of Json-files which contain a base64-coded field (Jsonpath to this field is $.data.content), and this field contains a XML-document. Decoding the field works as expected, so does the conversion from xml to json, and I'm able to write the content from this field to a file in a bucket in S3. But what I would like to do is to be able to replace the coded value for this field in the original file with the decoded/converted value in stead of writing the decoded/converted value to file. And after replacing the json-value then I can write the updated Json-file to a new S3 bucket. My process look like this at the moment, and works fine for getting the data to file, but it's missing the last part of replacing $.data.content with the decoded/converted data. So how can I do the last part? <pastedImage.png> The EvaluedJsonPath looks like this: <EvaluateJsonPath.png> The ReplaceText looks like this: <ReplaceText.png> The Base64EncodeContent looks like this: <Base64Decode.png> and finally, the CovertRecord looks like this: <ConvertRecord.png> This is a testfile for that I'm working with: { "header": { "dokumentidentifikator": null, "dokumentidentifikatorV2": "dcff985b-c652-4085-b8f1-45a2f4b6d150", "revisjonsnummer": 1, "dokumentnavn": "Engangsavgiftfastsettelse:55TEST661122334455:44BIL1:2017-10-20", "dokumenttype": "SKATTEMELDING_ENGANGSAVGIFT", "dokumenttilstand": "OPPRETTET", "gyldig": true, "gjelderInntektsaar": 2017, "gjelderPeriode": "2017_10", "gjelderPart": { "partsnummer": 5544332211, "identifiseringstype": "MASKINELL", "identifikator": null }, "opphavspart": { "partsnummer": 5544332211, "identifikator": null }, "kildereferanse": { "kildesystem": "ENGANGSAVGIFTFASTSETTELSE", "gruppe": "", "referanse": "aef147fb-8ce8-43ef-833b-7aa3bac1ece0", "tidspunkt": "2018-01-16T13:28:02.49+01:00" } }, "data": { "metadata": { "format": "ske:fastsetting:motorvogn:motorvognavgift:v1", "bytes": 4420, "mimeType": "application/xml", "sha1": "c0AowOsTdNdo6VufeSsZqTphc0Y=" }, "content": "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9InllcyI/Pgo8bW90b3J2b2duYXZnaWZ0IHhtbG5zPSJza2U6ZmFzdHNldHRpbmc6bW90b3J2b2duOm1vdG9ydm9nbmF2Z2lmdDp2MSI+CiAgICA8YXZnaWZ0c2xpbmplPgogICAgICAgIDxhdmdpZnRzYmVsb2VwPjU0Mjg5Ni4wMDwvYXZnaWZ0c2JlbG9lcD4KICAgICAgICA8YXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgICAgIDxzYWVyYXZnaWZ0VHlwZWtvZGU+QkI8L3NhZXJhdmdpZnRUeXBla29kZT4KICAgICAgICAgICAgPHNhZXJhdmdpZnRHcnVwcGVrb2RlPlg8L3NhZXJhdmdpZnRHcnVwcGVrb2RlPgogICAgICAgIDwvYXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgPGF2Z2lmdHNkYXRvPjIwMTctMTAtMjA8L2F2Z2lmdHNkYXRvPgogICAgPC9hdmdpZnRzbGluamU+CiAgICA8YmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICAgICAgPGtpZG51bW1lcj4xMDEwMTAxMDEwMTA8L2tpZG51bW1lcj4KICAgICAgICA8Zm9yZmFsbHNkYXRvPjIwMTctMTAtMjA8L2ZvcmZhbGxzZGF0bz4KICAgICAgICA8ZmFrdHVyYWRhdG8+MjAxNy0xMC0yMDwvZmFrdHVyYWRhdG8+CiAgICAgICAgPHRvdGFsQXZnaWZ0c2JlbG9lcD41NDI4OTYuMDA8L3RvdGFsQXZnaWZ0c2JlbG9lcD4KICAgIDwvYmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICA8bW90b3J2b2duYXZnaWZ0c3R5cGU+ZW5nYW5nc2F2Z2lmdDwvbW90b3J2b2duYXZnaWZ0c3R5cGU+CiAgICA8dGlkc3N0ZW1wZWw+MjAxOC0wMS0xNiswMTowMDwvdGlkc3N0ZW1wZWw+CiAgICA8Z3J1bm5sYWdGb3JNb3RvcnZvZ25hdmdpZnQ+CiAgICAgICAgPGtqb2VyaW5nZW5zQXJ0PjEwPC9ram9lcmluZ2Vuc0FydD4KICAgICAgICA8a2pvZXJldG9leT4KICAgICAgICAgICAgPGVpZXJza2FwUmVnaXN0cmVydD4yMDE3LTEwLTIwPC9laWVyc2thcFJlZ2lzdHJlcnQ+CiAgICAgICAgICAgIDxmb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4yMDE3PC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4KICAgICAgICAgICAgPGZvZXJzdGVSZWdpc3RyZXJpbmdzZGF0b0lOb3JnZT4yMDE3LTEwLTIwPC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2RhdG9JTm9yZ2U+CiAgICAgICAgICAgIDxram9lcmV0b2V5Z3J1cHBlPjEwMTwva2pvZXJldG9leWdydXBwZT4KICAgICAgICAgICAgPGxlbmdkZT4zOTY0PC9sZW5nZGU+CiAgICAgICAgICAgIDxtb3RvcmVmZmVrdD45NjwvbW90b3JlZmZla3Q+CiAgICAgICAgICAgIDxzbGFndm9sdW0+MTQzPC9zbGFndm9sdW0+CiAgICAgICAgICAgIDxkcml2c3RvZmY+QkVOU0lOPC9kcml2c3RvZmY+CiAgICAgICAgICAgIDxlZ2VudmVrdD4xNTE5PC9lZ2VudmVrdD4KICAgICAgICAgICAgPGVpZXI+CiAgICAgICAgICAgICAgICA8Zm9lZHNlbHNFbGxlckRudW1tZXI+MTEyMjMzNDQ1NTwvZm9lZHNlbHNFbGxlckRudW1tZXI+CiAgICAgICAgICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICAgICAgICAgICAgICA8bmF2bj5LTEFSQSBLVTwvbmF2bj4KICAgICAgICAgICAgPC9laWVyPgogICAgICAgICAgICA8dGlsbGF0dFRvdGFsdmVrdD4yMTY0PC90aWxsYXR0VG90YWx2ZWt0PgogICAgICAgICAgICA8aHlicmlkPm5laTwvaHlicmlkPgogICAgICAgICAgICA8Y28ydXRzbGlwcD4yNjg8L2NvMnV0c2xpcHA+CiAgICAgICAgICAgIDxub3h1dHNsaXBwPjU5LjQ8L25veHV0c2xpcHA+CiAgICAgICAgICAgIDxram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICAgICAgICAgIDxram9lcmV0b2V5VW5pa0lkZW50aWZpa2F0b3I+QUJDREVGR0hJSjwva2pvZXJldG9leVVuaWtJZGVudGlmaWthdG9yPgogICAgICAgICAgICAgICAgPGtqZW5uZW1lcmtlPjQ0QklMMTwva2plbm5lbWVya2U+CiAgICAgICAgICAgICAgICA8dW5kZXJzdGVsbHNudW1tZXI+VU5ERVJTVEVMTDQ0PC91bmRlcnN0ZWxsc251bW1lcj4KICAgICAgICAgICAgPC9ram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICA8L2tqb2VyZXRvZXk+CiAgICA8L2dydW5ubGFnRm9yTW90b3J2b2duYXZnaWZ0PgogICAgPGF2Z2lmdHNwbGlrdGlnPgogICAgICAgIDxmb2Vkc2Vsc0VsbGVyRG51bW1lcj4xMTIyMzM0NDU1PC9mb2Vkc2Vsc0VsbGVyRG51bW1lcj4KICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICA8L2F2Z2lmdHNwbGlrdGlnPgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzI8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjQ3NTMzNy4yMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5FZ2VudmVrdDwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NjA5NDUuNjQ8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3Q8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40MjEzLjI0PC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IEZyYXRyZWtrPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bSBGcmF0cmVrazwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40NzUzMzcuMjA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD42MDk0NS42NDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5Nb3RvcmVmZmVrdCBTdW08L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IFN1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NDIxMy4yNDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5TbGFndm9sdW0gU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlZyYWtwYW50PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4yNDAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PkJydWtzZnJhZHJhZyAwJTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGZvcmhhYW5kc2JlcmVnbmluZz5mYWxzZTwvZm9yaGFhbmRzYmVyZWduaW5nPgo8L21vdG9ydm9nbmF2Z2lmdD4K" }, "extension": null, "skjemaversjon": "v3_0" } Cheers, Bjørn ________________________________ Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den institusjon eller person den er rettet til og kan være belagt med lovbestemt taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt Skatteetaten. The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential information and may be legally protected from disclosure. If you are not the intended recipient of this message, please immediately delete the message and alert the Norwegian Tax Administration. <org.json.JSONObject.png><executegroovyscript.png>
