Joe, Awesome. +1 to "Use Chunked Encoding" as 'false' by default. I think for historical purposes of HTTP, if you know the size of your payload, you should explicitly assert it using content-length. Sending payloads with "normal" transfer encoding by default would be a great strategy.
>From a pragmatic point of view, if you know the size of your payload, there's really no reason to use chunked encoding. Having this feature property exposed at all is just a nicety, I guess. Adam On Fri, Jan 15, 2016 at 1:58 PM, Joe Percivall < joeperciv...@yahoo.com.invalid> wrote: > Adam, > > You are right and it looks like OkHttp could easily support it. > If you look at this line of OkHttp [1], you'll see that if the > contentLength is not set then it will use chunked and otherwise it won't. > If we added a property to InvokeHttp in which the user chooses to have > chunked or not then adjust the contentLength method of the RequestBody we > can enable the option. > I tried making a simple change in InvokeHttp to the RequestBody to just > implement the contentLength method with the correct value, combined with > updating the mime-type to "application/x-www-form-urlencoded" to send as > the content-type, and I successfully ran the template. I'll created a > ticket[2] and will have a patch up for the change shortly. > I'm going to assume that the default value for the "Use Chunked Encoding" > property will false. > [1] > https://github.com/square/okhttp/blob/parent-2.7.1/okhttp/src/main/java/com/squareup/okhttp/Call.java#L263[2] > https://issues.apache.org/jira/browse/NIFI-1405 > Joe - - - - - - > Joseph Percivall > linkedin.com/in/Percivall > e: joeperciv...@yahoo.com > > > > > > > On Friday, January 15, 2016 9:24 AM, Adam Taft <a...@adamtaft.com> wrote: > Joe, > > Just as a quick observation, this statement isn't completely accurate: > > > "... and can stream the contents instead of loading into memory" > > The original InvokeHTTP code (pre okhttp) explicitly set the content-length > header, because it was known (the flowfile payload content length is always > known). This does not, however, imply that the entire contents were loaded > into memory. The previous InvokeHTTP used the > #setFixedLengthStreamingMode(long), which is described as: > > "This method is used to enable streaming of a HTTP request body without > internal buffering, when the content length is known in advance." [1] > > HttpURLConnection doesn't need to buffer if the length is known in > advance. It's only when it doesn't know the length that it either needs to > buffer to determine it or use chunked encoding. > > I think it's important to be able to support non-chunked encoded POST > requests. There are many "legacy" (or even "broken") web services that > don't work with chunked encoding, obviously like in this case. > > Unfortunately, I don't recall that okhttp has similar direct support for > "fixed length streaming". It's probable that a custom implementation of > okhttp.RequestBody would need to be created to support this. [2] > > [1] > > https://docs.oracle.com/javase/8/docs/api/java/net/HttpURLConnection.html#setFixedLengthStreamingMode-long- > > [2] http://square.github.io/okhttp/3.x/okhttp/okhttp3/RequestBody.html > > > On Thu, Jan 14, 2016 at 10:29 PM, Joe Percivall < > joeperciv...@yahoo.com.invalid> wrote: > > > Hello Evan, > > > > Glad to hear you're enjoying NiFi! > > > > I was able to replicate your results so I dug in a bit and noticed in > > Wireshark that the "Transfer-Encoding" header for InvokeHttp was set to > > "chunked". When I tried using the same flag for curl it failed so I'm > > relatively confident that is the problem. Currently InvokeHttp requires > > using the chunk encoding for POST (primarily because you don't need to > know > > the content-length and can stream the contents instead of loading into > > memory). > > > > PostHttp does have a "Use Chunked Encoding" option which would solve your > > problem except that it doesn't work properly. PostHttp is using the > > "EntityTemplate" which streams the content so the content length will > never > > be implemented and thus it will alway use the chunked encoding. I > created a > > ticket for it [1]. > > > > > > Also as a note, when creating a template you have to either explicitly > > select the connections or not select anything and create a template for > the > > whole canvas (your template didn't have any connections). > > > > [1] https://issues.apache.org/jira/browse/NIFI-1396 > > > > Cheers, > > Joe > > - - - - - - > > Joseph Percivall > > linkedin.com/in/Percivall > > e: joeperciv...@yahoo.com > > > > > > > > On Thursday, January 14, 2016 8:07 PM, "yuchen....@thomsonreuters.com" < > > yuchen....@thomsonreuters.com> wrote: > > > > > > > > > > Hi Guys, > > > > Not sure if it is the correct way to raise issue by sending this email, > if > > not, let me know where the post the issue, thanks. > > > > We are using NiFi InvokeHttp processor to do POST to an webpage. > > URL: > > > http://www.hkexnews.hk/listedco/listconews/advancedsearch/search_active_main.aspx > > Request header: Content-Type: application/x-www-form-urlencoded > > POST Data: > > > txt_stock_code=24984&sel_DateOfReleaseFrom_y=2016&sel_DateOfReleaseFrom_m=01&sel_DateOfReleaseFrom_d=04&sel_DateOfReleaseTo_y=2016&sel_DateOfReleaseTo_m=01&sel_DateOfReleaseTo_d=11&IsFromNewList=False&sel_tier_1=-2&sel_tier_2_group=-2&sel_tier_2=-2 > > > > To make sure the request header and request body are correct, we use > > Fiddler to compose the post request. > > And the response show the request header and post data are correct. > > > > > > Attached file is the template we are using, it is working fine on version > > 0.3.0 > > > > But not on the latest version 0.4.1 > > > > So we suppose it is potential defect of the InvokeHttp processor in this > > version. > > We checked the source code and try to locate the issue, and found it is > > using com.squareup.okhttp.Request; to do the request, so we are not go > any > > further to dig the issue… > > Currently we are using Curl to do the POST as a workaround. > > > > Let me know your comments, thanks. > > > > Finally, NiFi is a great tool!!! You guys are awesome!!! > > > > Best Regards, > > Evan from Thomson Reuters > > >
