Wyllys, It may help to think of NiFi processors the same way you think about *nix command line tools — each tool has a very specific and limited responsibility and focus. Rather than continue to add generic features to each in order to cover all use cases, you combine the right tools in the necessary order to achieve your goal (i.e. cat, grep, sed, cut, etc.). No individual tool cares where the input data is coming from or where the output data is going; a critical idea which is part of the Flow Based Programming [1] design philosophy that NiFi models. Adding features increases complexity, configuration, and the opportunity for bugs or unexpected behavior, especially in edge cases.
The InvokeHTTP processor is concerned with transmitting flowfile attributes & content to a remote HTTP endpoint. Rather than include additional functionality to massage the content, we would recommend using an additional preceding processor to form the content into the expected format. I definitely understand the desire to add “just one thing” to make a specific use case easier, but we have to balance that with the maintainability and supportability of the framework as a whole moving forward. Hope this helps. Thanks. [1] https://nifi.apache.org/docs/nifi-docs/html/overview.html#the-core-concepts-of-nifi <https://nifi.apache.org/docs/nifi-docs/html/overview.html#the-core-concepts-of-nifi> Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Jun 19, 2019, at 7:23 AM, Wyllys Ingersoll > <wyllys.ingers...@keepertech.com> wrote: > > OK, that helps. I put AttributesToJSON and a ReplaceText processor in front > of my invokeHTTP and was able to get it to work as I wanted. > > One of the confusing issues is that InvokeHTTP does not have any > configuration that allows me to modify the flowfile content, which forces me > to use those additional processors to make sure the content is correct. It > might be a little more convenient if invokeHTTP had a couple of parameters to > allow for setting the content format so that the extra processors are not > necessary. > > > > On Tue, Jun 18, 2019 at 11:01 PM Andy LoPresto <alopre...@apache.org > <mailto:alopre...@apache.org>> wrote: > Wyllys, > > I am sorry but I am having a different experience when I try to reproduce the > scenario you describe. > > I set up a sample flow which uses an InvokeHTTP processor to send a flowfile > to a remote HTTP endpoint. That endpoint happens to be a disparate flow in > NiFi because it was easy to configure on the fly, but it could be any remote > web server. > > I can verify that when I send flowfile content, it is not transmitting the > flowfile attributes unless I explicitly configure it to. Please see below an > annotated excerpt from the nifi-app.log file and a link to the template I > exported with this flow. > > I think the issue may be one of terminology? Attributes and content are > separate in NiFi and flowfiles distinguish very cleanly between the two. Are > you referring to elements of the JSON body as “attributes”? In that case, you > would need to use a ReplaceText, EvaluateJsonPath, JoltTransformJson, or > other processor as Jerry suggested to extract/manipulate the JSON in the > flowfile content to the exact data you wish to send over HTTP before using > InvokeHTTP. If you want to do this but not lose the other content, you can > split the flow by dragging an additional “success” relationship from the > preceding processor to handle the two different flow behaviors. > > Template for flow (tested on NiFi 1.9.2): > https://gist.github.com/alopresto/0be1cf65ba70d5d36aeae1e6f2f60702 > <https://gist.github.com/alopresto/0be1cf65ba70d5d36aeae1e6f2f60702> > > Log output from flow: > https://gist.github.com/alopresto/b596372005032b9c80063a0652a955ea > <https://gist.github.com/alopresto/b596372005032b9c80063a0652a955ea> > > > Andy LoPresto > alopre...@apache.org <mailto:alopre...@apache.org> > alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On Jun 18, 2019, at 7:47 PM, Jerry Vinokurov <grapesmo...@gmail.com >> <mailto:grapesmo...@gmail.com>> wrote: >> >> Hi Wyllys, >> >> One way that I solve this problem in my work is to use the AttributesToJSON >> processor to form the body of the POST before sending it. Granted, that does >> override the body of the flowfile, so I'm not sure if that works for your >> specific case, but it does allow you to select which attributes you want to >> turn into JSON key/value pairs. For more complex formations I would suggest >> checking out the JOLT Transform processor, which can be quite powerful, if >> somewhat painful to work with. >> >> Jerry >> >> On Tue, Jun 18, 2019 at 9:49 PM Wyllys Ingersoll >> <wyllys.ingers...@keepertech.com <mailto:wyllys.ingers...@keepertech.com>> >> wrote: >> Andy - >> Yes, Im referring to the PUT and POST methods, in which case the >> processor just sends the entire flowfile as a JSON object in the message >> body. I'd prefer to either have the option to exclude some of the flow >> attributes or (even better) have the ability to craft my own message body. >> There are lots of instances where the receiver of the PUT or POST expects a >> particular structure that doesn't easily work with just a flat JSON-ified >> set of flow attributes. >> >> One example: >> We have a flow that has an authentication token as one of the attributes >> and a bunch of other key/value pairs used for other purposes. In the >> InvokeHTTP processor, I use the auth token attribute in an Authorization >> header by creating a dynamic attribute with the correct format >> ("Authorization: Bearer ${token}"). However, the recipient of the PUT/POST >> also expects the body of the request to be formatted a specific way and does >> not expect or want to see the auth token or some of the other unrelated >> information that ends up getting transmitted as part of the message body >> simply because they are flow attributes. So, if InvokeHTTP were able to >> exclude certain fields from the message body AND also allow the body of the >> message to be configurable into a structure other than just a flat >> dictionary of flow attributes, it would be much more powerful and useful. >> As it stands, I'm thinking I may have to develop a custom processor to get >> past this issue, which is not ideal at all. >> >> Thanks! >> Wyllys Ingersoll >> >> >> >> >> On Tue, Jun 18, 2019 at 8:34 PM Andy LoPresto <alopre...@apache.org >> <mailto:alopre...@apache.org>> wrote: >> Hi Wyllys, >> >> Sorry to hear you are having trouble with this processor. Can you please >> provide a more detailed example of an incoming flowfile and what your >> expected output is compared to what is currently happening? Based on my >> understanding, the flowfile attributes are only sent as request headers, and >> that can be controlled using the regular expression value of “Attributes to >> Send”. I believe only the flowfile content is sent as the request body when >> using PUT or POST. Thanks. >> >> >> Andy LoPresto >> alopre...@apache.org <mailto:alopre...@apache.org> >> alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >>> On Jun 18, 2019, at 3:01 PM, Wyllys Ingersoll >>> <wyllys.ingers...@keepertech.com <mailto:wyllys.ingers...@keepertech.com>> >>> wrote: >>> >>> >>> it would be nice to be able to exclude attributes from the message body >>> when doing a PUT or POST in the invokeHTTP processor. Currently, there's >>> no way to selectively choose which attributes go into the message body >>> without using a replaceText processor in front of it, but that completely >>> removes those attributes from being used later which is not always the >>> desirable. >>> >>> There are lots of situations where it would be nice to be able to use some >>> flow attributes just in the request header or for other parts of the >>> processor configuration without including them in the message body itself. >>> For example, taking an authentication token that is carried along as a flow >>> attribute so it can be used in an authorization header and NOT included in >>> the body of the request. >>> >>> In general, invokeHTTP needs to allow for more control over the format and >>> content of the message body being sent. >>> >>> -Wyllys Ingersoll >>> >> >> >> >> -- >> http://www.google.com/profiles/grapesmoker >> <http://www.google.com/profiles/grapesmoker>
