Re: Groovy script error, JsonSlurper
I will try this first thing when I get back on this system this evening. Thank you for your reply Matt. One followup: are there any imports i am overlooking to use json.each? I don't think so, but wanted to check now. Currently I import these json libraries, which I believe are baked into nifi groovy in my version, 1.16.3: import groovy.json.JsonSlurper import groovy.json.JsonOutput On Mon, May 8, 2023 at 10:38 PM Matt Burgess wrote: > I believe it's your "json.each { jsonObj ->" line, with a JSON object > it's going to return a key/value pair so try "json.each { k,v ->" > instead and use the key k and value v in your script. > > On Mon, May 8, 2023 at 9:32 PM James McMahon wrote: > > > > Hello. I have incoming data that is json. An example of one case looks > like this: > > > > {"id": "20230508215236_4447cd0a-9dca-47cb-90b1-6562cf34155a_Timer-Driven > Process Thread-9", > > "te": "0.9494", > > "diskusage": "0.2776125422110003.3 MB", > > "memory": 77, > > "cpu": 0.58, > > "host": "172.31.73.197/ip-172-31-73-197.ec2.internal", > > "temperature": "97", > > "macaddress": "f417ead3-4fa9-4cee-a14b-7172e9ecd3ea", > > "end": "61448816405795", > > "systemtime": "05/08/2023 16:52:36"} > > > > > > I wrote a Groovy script running in a nifi ExecuteScript processor to > tally the keys from the json, reporting the results as flowfile attributes. > I am getting this error: > > > > ExecuteScript[id=028b1d40-33a5-1766-b659-31b3faaf13f5] Error processing > json fields: groovy.lang.MissingMethodException: No signature of method: > Script9$_run_closure1$_closure2$_closure4.doCall() is applicable for > argument types: (Entry) values: > [id=20230508215236_4447cd0a-9dca-47cb-90b1-6562cf34155a_Timer-Driven > Process Thread-9] > > Possible solutions: doCall(java.lang.Object, java.lang.Object), > findAll(), findAll(), isCase(java.lang.Object), isCase(java.lang.Object) > > > > > > > > What is causing this error? Am I reading in the data incorrectly using > the JsonSlurper? > > > > > > > > Here is my code: > > > > import groovy.json.JsonSlurper > > import groovy.json.JsonOutput > > import org.apache.commons.io.IOUtils > > import java.nio.charset.StandardCharsets > > > > def originalFile = '' > > def keys = [] > > def tallyMap = [:] > > def topValuesMap = [:] > > def lineCount = 0 > > > > def ff = session.get() > > if (!ff) return > > try { > > session.read(ff, { inputStream -> > > def json = new > JsonSlurper().parseText(IOUtils.toString(inputStream, > StandardCharsets.UTF_8)) > > > > > > def keySet = new HashSet() > > > > if (json instanceof List) { > > keySet.addAll(((Map)json[0]).keySet()) > > } else { > > keySet.addAll(json.keySet()) > > } > > > > keys = keySet.toList() > > originalFile = ff.getAttribute('filename') > > > > > > json.each { jsonObj -> > > jsonObj.each { key, value -> > > if (value != null && !value.toString().trim().isEmpty()) > { > > tallyMap[key] = tallyMap.containsKey(key) ? > tallyMap[key] + 1 : 1 > > if (topValuesMap.containsKey(key)) { > > def valuesMap = topValuesMap[key] > > valuesMap[value] = valuesMap.containsKey(value) > ? valuesMap[value] + 1 : 1 > > topValuesMap[key] = valuesMap > > } else { > > topValuesMap[key] = [:].withDefault{ 0 > }.plus([value: 1]) > > } > > // Remove the "value: 1" entry from the topValuesMap > > topValuesMap[key].remove("value") > > } > > } > > } > > > > // Sort the topValuesMap for each key based on the frequency of > values > > topValuesMap.each { key, valuesMap -> > > topValuesMap[key] = valuesMap.sort{ -it.value }.take(10) > > } > > > > > > // Count the number of JSON records > > lineCount += json.size() > > } as InputStreamCallback) > > > > def tallyMapString = JsonOutput.toJson(tallyMap) > > def topValuesMapString = JsonOutput.toJson(topValuesMap) > > > > ff = session.putAttribute(ff, 'triage.json.file', originalFile) > > ff = session.putAttribute(ff, 'triage.json.fields', keys.join(",")) > > ff = session.putAttribute(ff, 'triage.json.tallyMap', tallyMapString) > > ff = session.putAttribute(ff, 'triage.json.topValuesMap', > topValuesMapString) > > ff = session.putAttribute(ff, 'triage.json.lineCount', > lineCount.toString()) > > session.transfer(ff, REL_SUCCESS) > > > > } catch (Exception e) { > > log.error('Error processing json fields', e) > > session.transfer(ff, REL_FAILURE) > > } > > > > > > >
Re: Groovy script error, JsonSlurper
I believe it's your "json.each { jsonObj ->" line, with a JSON object it's going to return a key/value pair so try "json.each { k,v ->" instead and use the key k and value v in your script. On Mon, May 8, 2023 at 9:32 PM James McMahon wrote: > > Hello. I have incoming data that is json. An example of one case looks like > this: > > {"id": "20230508215236_4447cd0a-9dca-47cb-90b1-6562cf34155a_Timer-Driven > Process Thread-9", > "te": "0.9494", > "diskusage": "0.2776125422110003.3 MB", > "memory": 77, > "cpu": 0.58, > "host": "172.31.73.197/ip-172-31-73-197.ec2.internal", > "temperature": "97", > "macaddress": "f417ead3-4fa9-4cee-a14b-7172e9ecd3ea", > "end": "61448816405795", > "systemtime": "05/08/2023 16:52:36"} > > > I wrote a Groovy script running in a nifi ExecuteScript processor to tally > the keys from the json, reporting the results as flowfile attributes. I am > getting this error: > > ExecuteScript[id=028b1d40-33a5-1766-b659-31b3faaf13f5] Error processing json > fields: groovy.lang.MissingMethodException: No signature of method: > Script9$_run_closure1$_closure2$_closure4.doCall() is applicable for argument > types: (Entry) values: > [id=20230508215236_4447cd0a-9dca-47cb-90b1-6562cf34155a_Timer-Driven Process > Thread-9] > Possible solutions: doCall(java.lang.Object, java.lang.Object), findAll(), > findAll(), isCase(java.lang.Object), isCase(java.lang.Object) > > > > What is causing this error? Am I reading in the data incorrectly using the > JsonSlurper? > > > > Here is my code: > > import groovy.json.JsonSlurper > import groovy.json.JsonOutput > import org.apache.commons.io.IOUtils > import java.nio.charset.StandardCharsets > > def originalFile = '' > def keys = [] > def tallyMap = [:] > def topValuesMap = [:] > def lineCount = 0 > > def ff = session.get() > if (!ff) return > try { > session.read(ff, { inputStream -> > def json = new JsonSlurper().parseText(IOUtils.toString(inputStream, > StandardCharsets.UTF_8)) > > > def keySet = new HashSet() > > if (json instanceof List) { > keySet.addAll(((Map)json[0]).keySet()) > } else { > keySet.addAll(json.keySet()) > } > > keys = keySet.toList() > originalFile = ff.getAttribute('filename') > > > json.each { jsonObj -> > jsonObj.each { key, value -> > if (value != null && !value.toString().trim().isEmpty()) { > tallyMap[key] = tallyMap.containsKey(key) ? tallyMap[key] > + 1 : 1 > if (topValuesMap.containsKey(key)) { > def valuesMap = topValuesMap[key] > valuesMap[value] = valuesMap.containsKey(value) ? > valuesMap[value] + 1 : 1 > topValuesMap[key] = valuesMap > } else { > topValuesMap[key] = [:].withDefault{ 0 }.plus([value: > 1]) > } > // Remove the "value: 1" entry from the topValuesMap > topValuesMap[key].remove("value") > } > } > } > > // Sort the topValuesMap for each key based on the frequency of values > topValuesMap.each { key, valuesMap -> > topValuesMap[key] = valuesMap.sort{ -it.value }.take(10) > } > > > // Count the number of JSON records > lineCount += json.size() > } as InputStreamCallback) > > def tallyMapString = JsonOutput.toJson(tallyMap) > def topValuesMapString = JsonOutput.toJson(topValuesMap) > > ff = session.putAttribute(ff, 'triage.json.file', originalFile) > ff = session.putAttribute(ff, 'triage.json.fields', keys.join(",")) > ff = session.putAttribute(ff, 'triage.json.tallyMap', tallyMapString) > ff = session.putAttribute(ff, 'triage.json.topValuesMap', > topValuesMapString) > ff = session.putAttribute(ff, 'triage.json.lineCount', > lineCount.toString()) > session.transfer(ff, REL_SUCCESS) > > } catch (Exception e) { > log.error('Error processing json fields', e) > session.transfer(ff, REL_FAILURE) > } > > >
Groovy script error, JsonSlurper
Hello. I have incoming data that is json. An example of one case looks like this: {"id": "20230508215236_4447cd0a-9dca-47cb-90b1-6562cf34155a_Timer-Driven Process Thread-9", "te": "0.9494", "diskusage": "0.2776125422110003.3 MB", "memory": 77, "cpu": 0.58, "host": "172.31.73.197/ip-172-31-73-197.ec2.internal", "temperature": "97", "macaddress": "f417ead3-4fa9-4cee-a14b-7172e9ecd3ea", "end": "61448816405795", "systemtime": "05/08/2023 16:52:36"} I wrote a Groovy script running in a nifi ExecuteScript processor to tally the keys from the json, reporting the results as flowfile attributes. I am getting this error: ExecuteScript[id=028b1d40-33a5-1766-b659-31b3faaf13f5] Error processing json fields: groovy.lang.MissingMethodException: No signature of method: Script9$_run_closure1$_closure2$_closure4.doCall() is applicable for argument types: (Entry) values: [id=20230508215236_4447cd0a-9dca-47cb-90b1-6562cf34155a_Timer-Driven Process Thread-9] Possible solutions: doCall(java.lang.Object, java.lang.Object), findAll(), findAll(), isCase(java.lang.Object), isCase(java.lang.Object) What is causing this error? Am I reading in the data incorrectly using the JsonSlurper? Here is my code: import groovy.json.JsonSlurper import groovy.json.JsonOutput import org.apache.commons.io.IOUtils import java.nio.charset.StandardCharsets def originalFile = '' def keys = [] def tallyMap = [:] def topValuesMap = [:] def lineCount = 0 def ff = session.get() if (!ff) return try { session.read(ff, { inputStream -> def json = new JsonSlurper().parseText(IOUtils.toString(inputStream, StandardCharsets.UTF_8)) def keySet = new HashSet() if (json instanceof List) { keySet.addAll(((Map)json[0]).keySet()) } else { keySet.addAll(json.keySet()) } keys = keySet.toList() originalFile = ff.getAttribute('filename') json.each { jsonObj -> jsonObj.each { key, value -> if (value != null && !value.toString().trim().isEmpty()) { tallyMap[key] = tallyMap.containsKey(key) ? tallyMap[key] + 1 : 1 if (topValuesMap.containsKey(key)) { def valuesMap = topValuesMap[key] valuesMap[value] = valuesMap.containsKey(value) ? valuesMap[value] + 1 : 1 topValuesMap[key] = valuesMap } else { topValuesMap[key] = [:].withDefault{ 0 }.plus([value: 1]) } // Remove the "value: 1" entry from the topValuesMap topValuesMap[key].remove("value") } } } // Sort the topValuesMap for each key based on the frequency of values topValuesMap.each { key, valuesMap -> topValuesMap[key] = valuesMap.sort{ -it.value }.take(10) } // Count the number of JSON records lineCount += json.size() } as InputStreamCallback) def tallyMapString = JsonOutput.toJson(tallyMap) def topValuesMapString = JsonOutput.toJson(topValuesMap) ff = session.putAttribute(ff, 'triage.json.file', originalFile) ff = session.putAttribute(ff, 'triage.json.fields', keys.join(",")) ff = session.putAttribute(ff, 'triage.json.tallyMap', tallyMapString) ff = session.putAttribute(ff, 'triage.json.topValuesMap', topValuesMapString) ff = session.putAttribute(ff, 'triage.json.lineCount', lineCount.toString()) session.transfer(ff, REL_SUCCESS) } catch (Exception e) { log.error('Error processing json fields', e) session.transfer(ff, REL_FAILURE) }