Re: Help on creating that flow that requires processing attributes in a flow content but need to preserve the original flow content
I’ve entered a Jira issue, NIFI-1660, to enhance the expression language to be
able to evaluate a Json path. I plan on submitting a pull request on it in the
next week or so.
On 3/22/16, 2:53 AM, "Conrad Crampton" wrote:
>My 2p.
>If the kaka.key value (very simple json), you could use UpdateAttribute and
>use some expression language - specifically the string manipulation functions
>to extract the part you want.
>I like the power or ExecuteProcessor by the way.
>
>And I agree, this community is phenomenally responsive and helpful.
>
>Regards
>Conrad
>
>
>
>
>On 21/03/2016, 18:38, "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)"
> wrote:
>
>>Thanks everyone. While I’m naturally disappointed that this doesn’t exist, I
>>am hyper charged about the responsiveness and enthusiasm of the NiFi
>>community!
>>
>>From: Matt Burgess mailto:mattyb...@gmail.com>>
>>Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
>>mailto:users@nifi.apache.org>>
>>Date: Monday, March 21, 2016 at 1:58 PM
>>To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
>>mailto:users@nifi.apache.org>>
>>Subject: Re: Help on creating that flow that requires processing attributes
>>in a flow content but need to preserve the original flow content
>>
>>One way (in NiFi 0.5.0+) is to use the ExecuteScript processor, which gives
>>you full control over the session and flowfile(s). For example if you had
>>JSON in your "kafka.key" attribute such as "{"data": {"myKey": "myValue"}}" ,
>>you could use the following Groovy script to parse out the value of the
>>'data.myKey' field:
>>
>>def flowfile = session.get()
>>if(!flowfile) return
>>def json = new
>>groovy.json.JsonSlurper().parseText(flowfile.getAttribute('kafka.key'))
>>flowfile = session.putAttribute(flowfile, 'myKey', json.data.myKey)
>>session.transfer(flowfile, REL_SUCCESS)
>>
>>
>>I put an example of this up as a Gist
>>(https://gist.github.com/mattyb149/478864017ec70d76f74f)
>>
>>A possible improvement could be to add a "jsonPath" function to Expression
>>Language, which could take any value (including an attribute) along with a
>>JSONPath expression to evaluate against it...
>>
>>Regards,
>>Matt
>>
>>On Mon, Mar 21, 2016 at 1:48 PM, McDermott, Chris Kevin (MSDU -
>>STaTS/StorefrontRemote)
>>mailto:chris.mcderm...@hpe.com>> wrote:
>>Joe,
>>
>>Thanks for the reply. I think I was not clear.
>>
>>The JSON I need to evaluate is in a FlowFile attribute (kafka.key) which I
>>need to be able to evaluate without modifying the original FlowFile content
>>(which was read from the Kafka topic). What I can’t figure out is how to
>>squirrel away the flowfile content so that I can write the value of the
>>kafka.key attribute to the FlowFile content, so that I can process it with
>>EvaluateJsonPath, and then read content I squirreled away back into the
>>FlowFile content. I considered using the the DistributedMapCache, but there
>>would be no guarantee what I added to the cache would still be there when I
>>needed it back.
>>
>>
>>
>>
>>On 3/21/16, 1:37 PM, "Joe Witt"
>>mailto:joe.w...@gmail.com>> wrote:
>>
>>>Chris,
>>>
>>>Sounds like you have the right flow in mind already. EvaluateJSONPath
>>>does not write content. It merely evaluates the given jsonpath
>>>expression against the content of the flowfile and if appropriate
>>>creates a flowfile attribute of what it finds.
>>>
>>>For example if you have JSON from Twitter you can use EvaluateJsonPath
>>>and add a property with a name
>>>'twitter.user' and a value of '$.user.name<http://user.name>'
>>>
>>>Once you run the tweets through each flow file will have an attribute
>>>called 'twitter.user' with the name found in the message. No
>>>manipulation of content at all. Just promotes things it finds to flow
>>>file attributes.
>>>
>>>Thanks
>>>Joe
>>>
>>>On Mon, Mar 21, 2016 at 1:34 PM, McDermott, Chris Kevin (MSDU -
>>>STaTS/StorefrontRemote)
>>>mailto:chris.mcderm...@hpe.com>> wrote:
>>>> What I need to do is read a file from Kafka. The Kafka key contains a
>>>> JSON string which I need to turn in FlowFile attributes while preserving
Re: Help on creating that flow that requires processing attributes in a flow content but need to preserve the original flow content
My 2p.
If the kaka.key value (very simple json), you could use UpdateAttribute and use
some expression language - specifically the string manipulation functions to
extract the part you want.
I like the power or ExecuteProcessor by the way.
And I agree, this community is phenomenally responsive and helpful.
Regards
Conrad
On 21/03/2016, 18:38, "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)"
wrote:
>Thanks everyone. While I’m naturally disappointed that this doesn’t exist, I
>am hyper charged about the responsiveness and enthusiasm of the NiFi community!
>
>From: Matt Burgess mailto:mattyb...@gmail.com>>
>Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
>mailto:users@nifi.apache.org>>
>Date: Monday, March 21, 2016 at 1:58 PM
>To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
>mailto:users@nifi.apache.org>>
>Subject: Re: Help on creating that flow that requires processing attributes in
>a flow content but need to preserve the original flow content
>
>One way (in NiFi 0.5.0+) is to use the ExecuteScript processor, which gives
>you full control over the session and flowfile(s). For example if you had
>JSON in your "kafka.key" attribute such as "{"data": {"myKey": "myValue"}}" ,
>you could use the following Groovy script to parse out the value of the
>'data.myKey' field:
>
>def flowfile = session.get()
>if(!flowfile) return
>def json = new
>groovy.json.JsonSlurper().parseText(flowfile.getAttribute('kafka.key'))
>flowfile = session.putAttribute(flowfile, 'myKey', json.data.myKey)
>session.transfer(flowfile, REL_SUCCESS)
>
>
>I put an example of this up as a Gist
>(https://gist.github.com/mattyb149/478864017ec70d76f74f)
>
>A possible improvement could be to add a "jsonPath" function to Expression
>Language, which could take any value (including an attribute) along with a
>JSONPath expression to evaluate against it...
>
>Regards,
>Matt
>
>On Mon, Mar 21, 2016 at 1:48 PM, McDermott, Chris Kevin (MSDU -
>STaTS/StorefrontRemote)
>mailto:chris.mcderm...@hpe.com>> wrote:
>Joe,
>
>Thanks for the reply. I think I was not clear.
>
>The JSON I need to evaluate is in a FlowFile attribute (kafka.key) which I
>need to be able to evaluate without modifying the original FlowFile content
>(which was read from the Kafka topic). What I can’t figure out is how to
>squirrel away the flowfile content so that I can write the value of the
>kafka.key attribute to the FlowFile content, so that I can process it with
>EvaluateJsonPath, and then read content I squirreled away back into the
>FlowFile content. I considered using the the DistributedMapCache, but there
>would be no guarantee what I added to the cache would still be there when I
>needed it back.
>
>
>
>
>On 3/21/16, 1:37 PM, "Joe Witt"
>mailto:joe.w...@gmail.com>> wrote:
>
>>Chris,
>>
>>Sounds like you have the right flow in mind already. EvaluateJSONPath
>>does not write content. It merely evaluates the given jsonpath
>>expression against the content of the flowfile and if appropriate
>>creates a flowfile attribute of what it finds.
>>
>>For example if you have JSON from Twitter you can use EvaluateJsonPath
>>and add a property with a name
>>'twitter.user' and a value of '$.user.name<http://user.name>'
>>
>>Once you run the tweets through each flow file will have an attribute
>>called 'twitter.user' with the name found in the message. No
>>manipulation of content at all. Just promotes things it finds to flow
>>file attributes.
>>
>>Thanks
>>Joe
>>
>>On Mon, Mar 21, 2016 at 1:34 PM, McDermott, Chris Kevin (MSDU -
>>STaTS/StorefrontRemote)
>>mailto:chris.mcderm...@hpe.com>> wrote:
>>> What I need to do is read a file from Kafka. The Kafka key contains a JSON
>>> string which I need to turn in FlowFile attributes while preserving the
>>> original FlowFile content. Obviously I can use EvaluteJsonPath but that
>>> necessitates replacing the FlowFile content with the kaka.key attribute,
>>> thus loosing the original FlowFile content. I feel like I’m missing
>>> something fundamental.
>
>
>
> ***This email originated outside SecureData***
>
>Click https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ== to report this
>email as spam.
SecureData, combating cyber threats
__
The information contained in this message or any of its attachments m
Re: Help on creating that flow that requires processing attributes in a flow content but need to preserve the original flow content
Thanks, Liam. I considered that but some of our files are rather large. It
may be a short term solution, though.
From: Lee Laim mailto:lee.l...@gmail.com>>
Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
mailto:users@nifi.apache.org>>
Date: Monday, March 21, 2016 at 3:17 PM
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
mailto:users@nifi.apache.org>>
Subject: Re: Help on creating that flow that requires processing attributes in
a flow content but need to preserve the original flow content
Chris,
Depending on the size* of the flowfile content, a combination of ExtractText
and ReplaceText might also work.
This is what I am picturing:
ExtractText the entire contents of the flowfile into a new attribute
flowfile.original.
ReplaceText with the ${kafka.key}. This will place the ${kafka.key} into the
output stream. Then EvaluateJSONPath with the kafka.key, return 'results'.
ExtractText 'results' into another attribute, and swap the original flowfile
contents out of the attributes and back into the output stream.
*Having large flowfiles in the attribute space may not be the most efficient
for processing and provenance. You may need to adjust the ExtractText property
Max Capture Group Length (default is 1024 )to accommodate this flow.
Thanks,
Lee
On Mon, Mar 21, 2016 at 12:38 PM, McDermott, Chris Kevin (MSDU -
STaTS/StorefrontRemote)
mailto:chris.mcderm...@hpe.com>> wrote:
Thanks everyone. While I’m naturally disappointed that this doesn’t exist, I
am hyper charged about the responsiveness and enthusiasm of the NiFi community!
From: Matt Burgess
mailto:mattyb...@gmail.com><mailto:mattyb...@gmail.com<mailto:mattyb...@gmail.com>>>
Reply-To:
"users@nifi.apache.org<mailto:users@nifi.apache.org><mailto:users@nifi.apache.org<mailto:users@nifi.apache.org>>"
mailto:users@nifi.apache.org><mailto:users@nifi.apache.org<mailto:users@nifi.apache.org>>>
Date: Monday, March 21, 2016 at 1:58 PM
To:
"users@nifi.apache.org<mailto:users@nifi.apache.org><mailto:users@nifi.apache.org<mailto:users@nifi.apache.org>>"
mailto:users@nifi.apache.org><mailto:users@nifi.apache.org<mailto:users@nifi.apache.org>>>
Subject: Re: Help on creating that flow that requires processing attributes in
a flow content but need to preserve the original flow content
One way (in NiFi 0.5.0+) is to use the ExecuteScript processor, which gives you
full control over the session and flowfile(s). For example if you had JSON in
your "kafka.key" attribute such as "{"data": {"myKey": "myValue"}}" , you could
use the following Groovy script to parse out the value of the 'data.myKey'
field:
def flowfile = session.get()
if(!flowfile) return
def json = new
groovy.json.JsonSlurper().parseText(flowfile.getAttribute('kafka.key'))
flowfile = session.putAttribute(flowfile, 'myKey', json.data.myKey)
session.transfer(flowfile, REL_SUCCESS)
I put an example of this up as a Gist
(https://gist.github.com/mattyb149/478864017ec70d76f74f)
A possible improvement could be to add a "jsonPath" function to Expression
Language, which could take any value (including an attribute) along with a
JSONPath expression to evaluate against it...
Regards,
Matt
On Mon, Mar 21, 2016 at 1:48 PM, McDermott, Chris Kevin (MSDU -
STaTS/StorefrontRemote)
mailto:chris.mcderm...@hpe.com><mailto:chris.mcderm...@hpe.com<mailto:chris.mcderm...@hpe.com>>>
wrote:
Joe,
Thanks for the reply. I think I was not clear.
The JSON I need to evaluate is in a FlowFile attribute (kafka.key) which I need
to be able to evaluate without modifying the original FlowFile content (which
was read from the Kafka topic). What I can’t figure out is how to squirrel
away the flowfile content so that I can write the value of the kafka.key
attribute to the FlowFile content, so that I can process it with
EvaluateJsonPath, and then read content I squirreled away back into the
FlowFile content. I considered using the the DistributedMapCache, but there
would be no guarantee what I added to the cache would still be there when I
needed it back.
On 3/21/16, 1:37 PM, "Joe Witt"
mailto:joe.w...@gmail.com><mailto:joe.w...@gmail.com<mailto:joe.w...@gmail.com>>>
wrote:
>Chris,
>
>Sounds like you have the right flow in mind already. EvaluateJSONPath
>does not write content. It merely evaluates the given jsonpath
>expression against the content of the flowfile and if appropriate
>creates a flowfile attribute of what it finds.
>
>For example if you have JSON from Twitter you can use EvaluateJsonPath
>and add a property with a name
>'twitter.user' and a value of '$.user.name&l
Re: Help on creating that flow that requires processing attributes in a flow content but need to preserve the original flow content
Chris,
Depending on the size* of the flowfile content, a combination of
ExtractText and ReplaceText might also work.
This is what I am picturing:
ExtractText the entire contents of the flowfile into a new attribute
flowfile.original.
ReplaceText with the ${kafka.key}. This will place the ${kafka.key} into
the output stream. Then EvaluateJSONPath with the kafka.key, return
'results'.
ExtractText 'results' into another attribute, and swap the original
flowfile contents out of the attributes and back into the output stream.
*Having large flowfiles in the attribute space may not be the most
efficient for processing and provenance. You may need to adjust the
ExtractText property Max Capture Group Length (default is 1024 )to
accommodate this flow.
Thanks,
Lee
On Mon, Mar 21, 2016 at 12:38 PM, McDermott, Chris Kevin (MSDU -
STaTS/StorefrontRemote) wrote:
> Thanks everyone. While I’m naturally disappointed that this doesn’t
> exist, I am hyper charged about the responsiveness and enthusiasm of the
> NiFi community!
>
> From: Matt Burgess mailto:mattyb...@gmail.com>>
> Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" <
> users@nifi.apache.org<mailto:users@nifi.apache.org>>
> Date: Monday, March 21, 2016 at 1:58 PM
> To: "users@nifi.apache.org<mailto:users@nifi.apache.org>" <
> users@nifi.apache.org<mailto:users@nifi.apache.org>>
> Subject: Re: Help on creating that flow that requires processing
> attributes in a flow content but need to preserve the original flow content
>
> One way (in NiFi 0.5.0+) is to use the ExecuteScript processor, which
> gives you full control over the session and flowfile(s). For example if
> you had JSON in your "kafka.key" attribute such as "{"data": {"myKey":
> "myValue"}}" , you could use the following Groovy script to parse out the
> value of the 'data.myKey' field:
>
> def flowfile = session.get()
> if(!flowfile) return
> def json = new
> groovy.json.JsonSlurper().parseText(flowfile.getAttribute('kafka.key'))
> flowfile = session.putAttribute(flowfile, 'myKey', json.data.myKey)
> session.transfer(flowfile, REL_SUCCESS)
>
>
> I put an example of this up as a Gist (
> https://gist.github.com/mattyb149/478864017ec70d76f74f)
>
> A possible improvement could be to add a "jsonPath" function to Expression
> Language, which could take any value (including an attribute) along with a
> JSONPath expression to evaluate against it...
>
> Regards,
> Matt
>
> On Mon, Mar 21, 2016 at 1:48 PM, McDermott, Chris Kevin (MSDU -
> STaTS/StorefrontRemote) chris.mcderm...@hpe.com>> wrote:
> Joe,
>
> Thanks for the reply. I think I was not clear.
>
> The JSON I need to evaluate is in a FlowFile attribute (kafka.key) which I
> need to be able to evaluate without modifying the original FlowFile content
> (which was read from the Kafka topic). What I can’t figure out is how to
> squirrel away the flowfile content so that I can write the value of the
> kafka.key attribute to the FlowFile content, so that I can process it with
> EvaluateJsonPath, and then read content I squirreled away back into the
> FlowFile content. I considered using the the DistributedMapCache, but there
> would be no guarantee what I added to the cache would still be there when I
> needed it back.
>
>
>
>
> On 3/21/16, 1:37 PM, "Joe Witt" joe.w...@gmail.com>> wrote:
>
> >Chris,
> >
> >Sounds like you have the right flow in mind already. EvaluateJSONPath
> >does not write content. It merely evaluates the given jsonpath
> >expression against the content of the flowfile and if appropriate
> >creates a flowfile attribute of what it finds.
> >
> >For example if you have JSON from Twitter you can use EvaluateJsonPath
> >and add a property with a name
> >'twitter.user' and a value of '$.user.name<http://user.name>'
> >
> >Once you run the tweets through each flow file will have an attribute
> >called 'twitter.user' with the name found in the message. No
> >manipulation of content at all. Just promotes things it finds to flow
> >file attributes.
> >
> >Thanks
> >Joe
> >
> >On Mon, Mar 21, 2016 at 1:34 PM, McDermott, Chris Kevin (MSDU -
> >STaTS/StorefrontRemote) chris.mcderm...@hpe.com>> wrote:
> >> What I need to do is read a file from Kafka. The Kafka key contains a
> JSON string which I need to turn in FlowFile attributes while preserving
> the original FlowFile content. Obviously I can use EvaluteJsonPath but
> that necessitates replacing the FlowFile content with the kaka.key
> attribute, thus loosing the original FlowFile content. I feel like I’m
> missing something fundamental.
>
>
Re: Help on creating that flow that requires processing attributes in a flow content but need to preserve the original flow content
Thanks everyone. While I’m naturally disappointed that this doesn’t exist, I
am hyper charged about the responsiveness and enthusiasm of the NiFi community!
From: Matt Burgess mailto:mattyb...@gmail.com>>
Reply-To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
mailto:users@nifi.apache.org>>
Date: Monday, March 21, 2016 at 1:58 PM
To: "users@nifi.apache.org<mailto:users@nifi.apache.org>"
mailto:users@nifi.apache.org>>
Subject: Re: Help on creating that flow that requires processing attributes in
a flow content but need to preserve the original flow content
One way (in NiFi 0.5.0+) is to use the ExecuteScript processor, which gives you
full control over the session and flowfile(s). For example if you had JSON in
your "kafka.key" attribute such as "{"data": {"myKey": "myValue"}}" , you could
use the following Groovy script to parse out the value of the 'data.myKey'
field:
def flowfile = session.get()
if(!flowfile) return
def json = new
groovy.json.JsonSlurper().parseText(flowfile.getAttribute('kafka.key'))
flowfile = session.putAttribute(flowfile, 'myKey', json.data.myKey)
session.transfer(flowfile, REL_SUCCESS)
I put an example of this up as a Gist
(https://gist.github.com/mattyb149/478864017ec70d76f74f)
A possible improvement could be to add a "jsonPath" function to Expression
Language, which could take any value (including an attribute) along with a
JSONPath expression to evaluate against it...
Regards,
Matt
On Mon, Mar 21, 2016 at 1:48 PM, McDermott, Chris Kevin (MSDU -
STaTS/StorefrontRemote)
mailto:chris.mcderm...@hpe.com>> wrote:
Joe,
Thanks for the reply. I think I was not clear.
The JSON I need to evaluate is in a FlowFile attribute (kafka.key) which I need
to be able to evaluate without modifying the original FlowFile content (which
was read from the Kafka topic). What I can’t figure out is how to squirrel
away the flowfile content so that I can write the value of the kafka.key
attribute to the FlowFile content, so that I can process it with
EvaluateJsonPath, and then read content I squirreled away back into the
FlowFile content. I considered using the the DistributedMapCache, but there
would be no guarantee what I added to the cache would still be there when I
needed it back.
On 3/21/16, 1:37 PM, "Joe Witt" mailto:joe.w...@gmail.com>>
wrote:
>Chris,
>
>Sounds like you have the right flow in mind already. EvaluateJSONPath
>does not write content. It merely evaluates the given jsonpath
>expression against the content of the flowfile and if appropriate
>creates a flowfile attribute of what it finds.
>
>For example if you have JSON from Twitter you can use EvaluateJsonPath
>and add a property with a name
>'twitter.user' and a value of '$.user.name<http://user.name>'
>
>Once you run the tweets through each flow file will have an attribute
>called 'twitter.user' with the name found in the message. No
>manipulation of content at all. Just promotes things it finds to flow
>file attributes.
>
>Thanks
>Joe
>
>On Mon, Mar 21, 2016 at 1:34 PM, McDermott, Chris Kevin (MSDU -
>STaTS/StorefrontRemote)
>mailto:chris.mcderm...@hpe.com>> wrote:
>> What I need to do is read a file from Kafka. The Kafka key contains a JSON
>> string which I need to turn in FlowFile attributes while preserving the
>> original FlowFile content. Obviously I can use EvaluteJsonPath but that
>> necessitates replacing the FlowFile content with the kaka.key attribute,
>> thus loosing the original FlowFile content. I feel like I’m missing
>> something fundamental.
Re: Help on creating that flow that requires processing attributes in a flow content but need to preserve the original flow content
One way (in NiFi 0.5.0+) is to use the ExecuteScript processor, which gives
you full control over the session and flowfile(s). For example if you had
JSON in your "kafka.key" attribute such as "{"data": {"myKey": "myValue"}}"
, you could use the following Groovy script to parse out the value of the
'data.myKey' field:
def flowfile = session.get()
if(!flowfile) return
def json = new
groovy.json.JsonSlurper().parseText(flowfile.getAttribute('kafka.key'))
flowfile = session.putAttribute(flowfile, 'myKey', json.data.myKey)
session.transfer(flowfile, REL_SUCCESS)
I put an example of this up as a Gist (
https://gist.github.com/mattyb149/478864017ec70d76f74f)
A possible improvement could be to add a "jsonPath" function to Expression
Language, which could take any value (including an attribute) along with a
JSONPath expression to evaluate against it...
Regards,
Matt
On Mon, Mar 21, 2016 at 1:48 PM, McDermott, Chris Kevin (MSDU -
STaTS/StorefrontRemote) wrote:
> Joe,
>
> Thanks for the reply. I think I was not clear.
>
> The JSON I need to evaluate is in a FlowFile attribute (kafka.key) which I
> need to be able to evaluate without modifying the original FlowFile content
> (which was read from the Kafka topic). What I can’t figure out is how to
> squirrel away the flowfile content so that I can write the value of the
> kafka.key attribute to the FlowFile content, so that I can process it with
> EvaluateJsonPath, and then read content I squirreled away back into the
> FlowFile content. I considered using the the DistributedMapCache, but there
> would be no guarantee what I added to the cache would still be there when I
> needed it back.
>
>
>
>
> On 3/21/16, 1:37 PM, "Joe Witt" wrote:
>
> >Chris,
> >
> >Sounds like you have the right flow in mind already. EvaluateJSONPath
> >does not write content. It merely evaluates the given jsonpath
> >expression against the content of the flowfile and if appropriate
> >creates a flowfile attribute of what it finds.
> >
> >For example if you have JSON from Twitter you can use EvaluateJsonPath
> >and add a property with a name
> >'twitter.user' and a value of '$.user.name'
> >
> >Once you run the tweets through each flow file will have an attribute
> >called 'twitter.user' with the name found in the message. No
> >manipulation of content at all. Just promotes things it finds to flow
> >file attributes.
> >
> >Thanks
> >Joe
> >
> >On Mon, Mar 21, 2016 at 1:34 PM, McDermott, Chris Kevin (MSDU -
> >STaTS/StorefrontRemote) wrote:
> >> What I need to do is read a file from Kafka. The Kafka key contains a
> JSON string which I need to turn in FlowFile attributes while preserving
> the original FlowFile content. Obviously I can use EvaluteJsonPath but
> that necessitates replacing the FlowFile content with the kaka.key
> attribute, thus loosing the original FlowFile content. I feel like I’m
> missing something fundamental.
>
Re: Help on creating that flow that requires processing attributes in a flow content but need to preserve the original flow content
Chris - also you were clear. I was just too quick to reply and didn't read carefully. On Mon, Mar 21, 2016 at 1:53 PM, Mark Payne wrote: > Chris, > > Unfortunately, at this time, the EvaluateJsonPath requires that the JSON to > evaluate be the content of the FlowFIle. > There already does exist a ticket [1] that would allow you to specify an > attribute to use as the JSON instead of > requiring that it be the content only. Unfortunately, this has not yet been > implemented. > > Thanks > -Mark > > [1] https://issues.apache.org/jira/browse/NIFI-1567 > > >> On Mar 21, 2016, at 1:48 PM, McDermott, Chris Kevin (MSDU - >> STaTS/StorefrontRemote) wrote: >> >> Joe, >> >> Thanks for the reply. I think I was not clear. >> >> The JSON I need to evaluate is in a FlowFile attribute (kafka.key) which I >> need to be able to evaluate without modifying the original FlowFile content >> (which was read from the Kafka topic). What I can’t figure out is how to >> squirrel away the flowfile content so that I can write the value of the >> kafka.key attribute to the FlowFile content, so that I can process it with >> EvaluateJsonPath, and then read content I squirreled away back into the >> FlowFile content. I considered using the the DistributedMapCache, but there >> would be no guarantee what I added to the cache would still be there when I >> needed it back. >> >> >> >> >> On 3/21/16, 1:37 PM, "Joe Witt" wrote: >> >>> Chris, >>> >>> Sounds like you have the right flow in mind already. EvaluateJSONPath >>> does not write content. It merely evaluates the given jsonpath >>> expression against the content of the flowfile and if appropriate >>> creates a flowfile attribute of what it finds. >>> >>> For example if you have JSON from Twitter you can use EvaluateJsonPath >>> and add a property with a name >>> 'twitter.user' and a value of '$.user.name' >>> >>> Once you run the tweets through each flow file will have an attribute >>> called 'twitter.user' with the name found in the message. No >>> manipulation of content at all. Just promotes things it finds to flow >>> file attributes. >>> >>> Thanks >>> Joe >>> >>> On Mon, Mar 21, 2016 at 1:34 PM, McDermott, Chris Kevin (MSDU - >>> STaTS/StorefrontRemote) wrote: What I need to do is read a file from Kafka. The Kafka key contains a JSON string which I need to turn in FlowFile attributes while preserving the original FlowFile content. Obviously I can use EvaluteJsonPath but that necessitates replacing the FlowFile content with the kaka.key attribute, thus loosing the original FlowFile content. I feel like I’m missing something fundamental. >
Re: Help on creating that flow that requires processing attributes in a flow content but need to preserve the original flow content
Chris, Unfortunately, at this time, the EvaluateJsonPath requires that the JSON to evaluate be the content of the FlowFIle. There already does exist a ticket [1] that would allow you to specify an attribute to use as the JSON instead of requiring that it be the content only. Unfortunately, this has not yet been implemented. Thanks -Mark [1] https://issues.apache.org/jira/browse/NIFI-1567 > On Mar 21, 2016, at 1:48 PM, McDermott, Chris Kevin (MSDU - > STaTS/StorefrontRemote) wrote: > > Joe, > > Thanks for the reply. I think I was not clear. > > The JSON I need to evaluate is in a FlowFile attribute (kafka.key) which I > need to be able to evaluate without modifying the original FlowFile content > (which was read from the Kafka topic). What I can’t figure out is how to > squirrel away the flowfile content so that I can write the value of the > kafka.key attribute to the FlowFile content, so that I can process it with > EvaluateJsonPath, and then read content I squirreled away back into the > FlowFile content. I considered using the the DistributedMapCache, but there > would be no guarantee what I added to the cache would still be there when I > needed it back. > > > > > On 3/21/16, 1:37 PM, "Joe Witt" wrote: > >> Chris, >> >> Sounds like you have the right flow in mind already. EvaluateJSONPath >> does not write content. It merely evaluates the given jsonpath >> expression against the content of the flowfile and if appropriate >> creates a flowfile attribute of what it finds. >> >> For example if you have JSON from Twitter you can use EvaluateJsonPath >> and add a property with a name >> 'twitter.user' and a value of '$.user.name' >> >> Once you run the tweets through each flow file will have an attribute >> called 'twitter.user' with the name found in the message. No >> manipulation of content at all. Just promotes things it finds to flow >> file attributes. >> >> Thanks >> Joe >> >> On Mon, Mar 21, 2016 at 1:34 PM, McDermott, Chris Kevin (MSDU - >> STaTS/StorefrontRemote) wrote: >>> What I need to do is read a file from Kafka. The Kafka key contains a JSON >>> string which I need to turn in FlowFile attributes while preserving the >>> original FlowFile content. Obviously I can use EvaluteJsonPath but that >>> necessitates replacing the FlowFile content with the kaka.key attribute, >>> thus loosing the original FlowFile content. I feel like I’m missing >>> something fundamental.
Re: Help on creating that flow that requires processing attributes in a flow content but need to preserve the original flow content
Joe, Thanks for the reply. I think I was not clear. The JSON I need to evaluate is in a FlowFile attribute (kafka.key) which I need to be able to evaluate without modifying the original FlowFile content (which was read from the Kafka topic). What I can’t figure out is how to squirrel away the flowfile content so that I can write the value of the kafka.key attribute to the FlowFile content, so that I can process it with EvaluateJsonPath, and then read content I squirreled away back into the FlowFile content. I considered using the the DistributedMapCache, but there would be no guarantee what I added to the cache would still be there when I needed it back. On 3/21/16, 1:37 PM, "Joe Witt" wrote: >Chris, > >Sounds like you have the right flow in mind already. EvaluateJSONPath >does not write content. It merely evaluates the given jsonpath >expression against the content of the flowfile and if appropriate >creates a flowfile attribute of what it finds. > >For example if you have JSON from Twitter you can use EvaluateJsonPath >and add a property with a name >'twitter.user' and a value of '$.user.name' > >Once you run the tweets through each flow file will have an attribute >called 'twitter.user' with the name found in the message. No >manipulation of content at all. Just promotes things it finds to flow >file attributes. > >Thanks >Joe > >On Mon, Mar 21, 2016 at 1:34 PM, McDermott, Chris Kevin (MSDU - >STaTS/StorefrontRemote) wrote: >> What I need to do is read a file from Kafka. The Kafka key contains a JSON >> string which I need to turn in FlowFile attributes while preserving the >> original FlowFile content. Obviously I can use EvaluteJsonPath but that >> necessitates replacing the FlowFile content with the kaka.key attribute, >> thus loosing the original FlowFile content. I feel like I’m missing >> something fundamental.
Re: Help on creating that flow that requires processing attributes in a flow content but need to preserve the original flow content
Hello Chris, The EvaluateJsonPath processor has the property "Destination" which gives you the option to send it either to the FlowFile content or a FlowFile attribute. Selecting "flowfile-attribute" will place the value in the "kafka.key" attribute of the FlowFile. You can find documentation for EvaluateJsonPath here[1]. [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.EvaluateJsonPath/index.html Hope your NiFi use is going well, Joe - - - - - - Joseph Percivall linkedin.com/in/Percivall e: joeperciv...@yahoo.com On Monday, March 21, 2016 1:34 PM, "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)" wrote: What I need to do is read a file from Kafka. The Kafka key contains a JSON string which I need to turn in FlowFile attributes while preserving the original FlowFile content. Obviously I can use EvaluteJsonPath but that necessitates replacing the FlowFile content with the kaka.key attribute, thus loosing the original FlowFile content. I feel like I’m missing something fundamental.
Re: Help on creating that flow that requires processing attributes in a flow content but need to preserve the original flow content
Chris, Sounds like you have the right flow in mind already. EvaluateJSONPath does not write content. It merely evaluates the given jsonpath expression against the content of the flowfile and if appropriate creates a flowfile attribute of what it finds. For example if you have JSON from Twitter you can use EvaluateJsonPath and add a property with a name 'twitter.user' and a value of '$.user.name' Once you run the tweets through each flow file will have an attribute called 'twitter.user' with the name found in the message. No manipulation of content at all. Just promotes things it finds to flow file attributes. Thanks Joe On Mon, Mar 21, 2016 at 1:34 PM, McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote) wrote: > What I need to do is read a file from Kafka. The Kafka key contains a JSON > string which I need to turn in FlowFile attributes while preserving the > original FlowFile content. Obviously I can use EvaluteJsonPath but that > necessitates replacing the FlowFile content with the kaka.key attribute, thus > loosing the original FlowFile content. I feel like I’m missing something > fundamental.
Help on creating that flow that requires processing attributes in a flow content but need to preserve the original flow content
What I need to do is read a file from Kafka. The Kafka key contains a JSON string which I need to turn in FlowFile attributes while preserving the original FlowFile content. Obviously I can use EvaluteJsonPath but that necessitates replacing the FlowFile content with the kaka.key attribute, thus loosing the original FlowFile content. I feel like I’m missing something fundamental.
