[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013 --- Comment #16 from Felix Schumacher --- commit 66e492113bde973ce5d0ed163efb5aa61efe3145 AuthorDate: Fri Jan 15 18:54:12 2021 +0100 Remove leftover log message from debug session Instead of logging at info level, log it with debug, only and remove the unnecessary creation of a RuntimeException. Cleanup after commit bb0c4557e10278f334844f8c1af873b708981b08 Bugzilla Id: 65013 --- .../main/java/org/apache/jmeter/protocol/http/curl/BasicCurlParser.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- You are receiving this mail because: You are the assignee for the bug.
[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013
eR@SeR changed:
What|Removed |Added
Status|NEEDINFO|NEW
--- Comment #15 from eR@SeR ---
> However, some commonly used extensions like .bmp .pdf .doc .docx .xls .xlsx
> .rar .zip etc. are still not recognized and generated MIME type is still
> application/octet-stream. You can check "Common MIME types" list on
> https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types
> and add all mentioned MIME types.
I checked all extensions from the MIME types list and almost all were properly
detected. Some are not, some differ a little bit, but I guess that is fine.
Nice improvement though :)
> 1. If parameter starts with @ like "@somevalue", form-data parameter is put
> inside Files upload tab instead to remain in Parameters tab.
> 2. If same header or parameter name with same or different value is added,
> only last header or parameter is generated. All headers or parameters should
> be added even if they are duplicated.
> 3. If URL, header or parameter value is blank, cURL is not imported.
Those three are fixed except if URL is blank. As you mentioned, that is already
reported in https://bz.apache.org/bugzilla/show_bug.cgi?id=64636
(In reply to Felix Schumacher from comment #13)
> Created attachment 37646 [details]
> Fixes for multi-header and -parameter and mime detection
>
> Another iteration on the remaining problems.
>
> curl can do a lot of stuff when specifying parameters, like
> 'file=@filename;type=text/html' (which has been partly supported already) or
> 'param="something with spaces";headers=\"X-something: one\"'
>
> The latter example is not possible right now, but a small step has been
> taken in that direction. Missing is also support for
> 'file=@"some/file";type=text/xml' that is a filename which was quoted.
Probably, the following parameter and file parameter example that I've found
might be related, so please confirm (or fix if not):
curl --location --request POST 'http://example.com/api/somepath' \
--header 'Accept-Language: it-IT' \
--header 'HTTP_X_FORWARDED_FOR: 127.0.0.1' \
--form 'FrontsideImage=@"/C:/Users/someuser/Desktop/Desktop.jpg"' \
--form 'BacksideImage=@"/C:/Users/someuser/Desktop/Desktop.jpg"' \
--form 'ApplicationTypeId=""' \
--form 'Country="ITA"' \
--form 'DocumentType="ID_CARD"' \
--form 'parameterWithAllSymbols="testTEST
~`!@#$%^&*()_+1234567890[]{};'\''\\:\"|,./<>?"' \
--form 'fileNameWithAllAllowedSymbolsByWin=@"/C:/Users/someuser/Desktop/Desktop
0123456789 `~!@#$&()_+{}[];'\''..jpg"'
Actual chars typed in Postman:
parameterWithAllSymbols = testTEST ~`!@#$%^&*()_+1234567890[]{};'\:"|,./<>?
fileNameWithAllAllowedSymbolsByWin = Desktop 0123456789 `~!@#$&()_+{}[];'..jpg
It looks like that ' and ; are problematic ones. If they are removed, cURL is
properly imported.
Interesting part was if ; is put at the end of string, but without ' character
in parameterWithAllSymbols value, cURL is properly imported, but the same for '
is not working.
You said 'missing support'. Is there a workaround to make exceptions by
yourself? Maybe simple find and replace or regex can do the trick?
I noticed that ' is generated in Postman as '\''
Can find and replace fix it?
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013 Felix Schumacher changed: What|Removed |Added Status|NEW |NEEDINFO --- Comment #14 from Felix Schumacher --- Could you try next nightly or build from trunk and report back, whether your issues are solved? Remember completely empty parameters like in "curl ''" are still not supported. But I think a lot of your reported ones are fixed. commit f045cf5e1604c68b7d43986fe9bd82a102fa2b76 Author: Felix Schumacher AuthorDate: Wed Dec 23 10:45:44 2020 +0100 Sending mime type with parameter throws IllegalArgumentException More stuff from that bugzilla entry: * Better distinguishing between a quoted filename and a quoted name, that contains an @ char * Allow more than one header and parameter with the same name to be added * Simplify the code in ParseCurlCommandAction by moving some of the logic in extra classes What is still missing are curl specialities like --form file=@"file with; semicolon or space";type=text/something --form param="red green blue";headers="X-Something: \"yeah; something\"" Bugzilla Id: 65024 --- src/protocol/build.gradle.kts | 8 +++ .../jmeter/protocol/http/curl/ArgumentHolder.java | 55 +++ .../jmeter/protocol/http/curl/BasicCurlParser.java | 51 +- .../protocol/http/curl/FileArgumentHolder.java | 80 ++ .../protocol/http/curl/StringArgumentHolder.java | 74 .../http/gui/action/ParseCurlCommandAction.java| 57 +++ .../apache/jmeter/curl/BasicCurlParserTest.java| 72 ++- 7 files changed, 334 insertions(+), 63 deletions(-) commit 9d2ba68c16a079c4e7156d847ed9acb7a6d4bb88 Author: Felix Schumacher AuthorDate: Wed Dec 23 17:12:47 2020 +0100 Sending mime type with parameter throws IllegalArgumentException More stuff from that bugzilla entry: * Allow empty headers to be added Bugzilla Id: 65024 --- .../apache/jmeter/protocol/http/curl/BasicCurlParser.java | 14 ++ .../java/org/apache/jmeter/curl/BasicCurlParserTest.java | 10 ++ 2 files changed, 20 insertions(+), 4 deletions(-) -- You are receiving this mail because: You are the assignee for the bug.
[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013 Felix Schumacher changed: What|Removed |Added Attachment #37644|0 |1 is obsolete|| Attachment #37645|0 |1 is obsolete|| --- Comment #13 from Felix Schumacher --- Created attachment 37646 --> https://bz.apache.org/bugzilla/attachment.cgi?id=37646=edit Fixes for multi-header and -parameter and mime detection Another iteration on the remaining problems. curl can do a lot of stuff when specifying parameters, like 'file=@filename;type=text/html' (which has been partly supported already) or 'param="something with spaces";headers=\"X-something: one\"' The latter example is not possible right now, but a small step has been taken in that direction. Missing is also support for 'file=@"some/file";type=text/xml' that is a filename which was quoted. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013 --- Comment #12 from Felix Schumacher --- Created attachment 37645 --> https://bz.apache.org/bugzilla/attachment.cgi?id=37645=edit Fixes for multi-header and -parameter and mime detection Now, it tries to differentiate between "@value" and @"value". -- You are receiving this mail because: You are the assignee for the bug.
[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013 --- Comment #11 from eR@SeR --- (In reply to Felix Schumacher from comment #10) > The currently used parser for CLI options can not cope with empty > parameters. So, I think it is the same report. All right. > That changes the story :) I missed that one. curl does indeed differentiate > between name="@..." and name=@"...". Nice :) -- You are receiving this mail because: You are the assignee for the bug.
[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013 --- Comment #10 from Felix Schumacher --- (In reply to eR@SeR from comment #8) > (In reply to Felix Schumacher from comment #5) > > For the third bug report part, you could look at > > https://bz.apache.org/bugzilla/show_bug.cgi?id=64636 which is probably the > > same as your report. > > cURL is not the same as I reported, but if > https://bz.apache.org/bugzilla/show_bug.cgi?id=64636 gets fixed, might be > fixed for reported one. Please note that same happens if parameter or header > values are blank. The currently used parser for CLI options can not cope with empty parameters. So, I think it is the same report. > > (In reply to Felix Schumacher from comment #6) > > For the second part (parameters which values start with @ landing in files > > upload tab), I believe, JMeter is doing the correct thing here. > > > > The curl man page states in the '--form' section: "... To force the > > 'content' part to be a file, prefix the file name with an @ sign. ..." > > > > If you want to send a value starting with an @ sign, you probably have to > > use --form-string as parameter. The curl man page for '-form-string' says: > > "... Similar to -F, --form except that the value string for the named > > parameter is used literally. Leading '@' and '<' characters, and the > > ';type=' string in > > the value have no special meaning. ..." > > > > Maybe postman tries the value in the parameter, before it decides, whether > > this is a file or not. JMeter can't really decide this correctly, as the > > file has not to be present on creation of the test plan. It has to be > > present when the test plan is being run. > > > > I am not sure, what to do about this, but it might be a good idea to discuss > > this on the mailing list and/or put this into its own bug report, as it will > > change the behaviour of detection. > > You might missed difference between: > > --form 'DocumentType="@ID_CARD"' \ > --form 'BacksideImage=@"/C:/Users/someuser/Desktop/Desktop.jpg"' \ > > @ in DocumentType parameter is inside quotes, but in BacksideImage is before > quotes. Does this change anything? That changes the story :) I missed that one. curl does indeed differentiate between name="@..." and name=@"...". -- You are receiving this mail because: You are the assignee for the bug.
[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013 Felix Schumacher changed: What|Removed |Added Attachment #37643|0 |1 is obsolete|| --- Comment #9 from Felix Schumacher --- Created attachment 37644 --> https://bz.apache.org/bugzilla/attachment.cgi?id=37644=edit Fixes for multi-header and -parameter and mime detection Now with test cases -- You are receiving this mail because: You are the assignee for the bug.
[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013 --- Comment #8 from eR@SeR --- (In reply to Felix Schumacher from comment #5) > For the third bug report part, you could look at > https://bz.apache.org/bugzilla/show_bug.cgi?id=64636 which is probably the > same as your report. cURL is not the same as I reported, but if https://bz.apache.org/bugzilla/show_bug.cgi?id=64636 gets fixed, might be fixed for reported one. Please note that same happens if parameter or header values are blank. (In reply to Felix Schumacher from comment #6) > For the second part (parameters which values start with @ landing in files > upload tab), I believe, JMeter is doing the correct thing here. > > The curl man page states in the '--form' section: "... To force the > 'content' part to be a file, prefix the file name with an @ sign. ..." > > If you want to send a value starting with an @ sign, you probably have to > use --form-string as parameter. The curl man page for '-form-string' says: > "... Similar to -F, --form except that the value string for the named > parameter is used literally. Leading '@' and '<' characters, and the > ';type=' string in > the value have no special meaning. ..." > > Maybe postman tries the value in the parameter, before it decides, whether > this is a file or not. JMeter can't really decide this correctly, as the > file has not to be present on creation of the test plan. It has to be > present when the test plan is being run. > > I am not sure, what to do about this, but it might be a good idea to discuss > this on the mailing list and/or put this into its own bug report, as it will > change the behaviour of detection. You might missed difference between: --form 'DocumentType="@ID_CARD"' \ --form 'BacksideImage=@"/C:/Users/someuser/Desktop/Desktop.jpg"' \ @ in DocumentType parameter is inside quotes, but in BacksideImage is before quotes. Does this change anything? -- You are receiving this mail because: You are the assignee for the bug.
[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013 --- Comment #7 from Felix Schumacher --- Created attachment 37643 --> https://bz.apache.org/bugzilla/attachment.cgi?id=37643=edit Fixes for multi-header and -parameter and mime detection With this path multiple headers/parameters with the same name will get added to the generated request. It will also change the detection method for mimetypes to be hopefully a bit more precise. It uses Apache Tika and tries to look at the content of the file (if it can), as well as the extension of the filename. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013 --- Comment #6 from Felix Schumacher --- For the second part (parameters which values start with @ landing in files upload tab), I believe, JMeter is doing the correct thing here. The curl man page states in the '--form' section: "... To force the 'content' part to be a file, prefix the file name with an @ sign. ..." If you want to send a value starting with an @ sign, you probably have to use --form-string as parameter. The curl man page for '-form-string' says: "... Similar to -F, --form except that the value string for the named parameter is used literally. Leading '@' and '<' characters, and the ';type=' string in the value have no special meaning. ..." Maybe postman tries the value in the parameter, before it decides, whether this is a file or not. JMeter can't really decide this correctly, as the file has not to be present on creation of the test plan. It has to be present when the test plan is being run. I am not sure, what to do about this, but it might be a good idea to discuss this on the mailing list and/or put this into its own bug report, as it will change the behaviour of detection. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013 --- Comment #5 from Felix Schumacher --- For the third bug report part, you could look at https://bz.apache.org/bugzilla/show_bug.cgi?id=64636 which is probably the same as your report. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013 eR@SeR changed: What|Removed |Added Status|NEEDINFO|NEW --- Comment #4 from eR@SeR --- Hi, Quotes around parameter values are not generated anymore and if file extension is .png .jpg .jpeg .gif .avi (might be more) MIME type is properly recognized. However, some commonly used extensions like .bmp .pdf .doc .docx .xls .xlsx .rar .zip etc. are still not recognized and generated MIME type is still application/octet-stream. You can check "Common MIME types" list on https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types and add all mentioned MIME types. Besides this, I've found a few more findings that might be bugs: 1. If parameter starts with @ like "@somevalue", form-data parameter is put inside Files upload tab instead to remain in Parameters tab. 2. If same header or parameter name with same or different value is added, only last header or parameter is generated. All headers or parameters should be added even if they are duplicated. cURL example: curl --location --request POST 'http://example.com/api/somepath' \ --header 'Accept-Language: it-IT' \ --header 'HTTP_X_FORWARDED_FOR: 127.0.0.1' \ --header 'HTTP_X_FORWARDED_FOR: 127.0.0.2' \ --form 'FrontsideImage=@"/C:/Users/someuser/Desktop/Desktop.avi"' \ --form 'BacksideImage=@"/C:/Users/someuser/Desktop/Desktop.jpg"' \ --form 'BacksideImage=@"/C:/Users/someuser/Desktop/Desktop.jpeg"' \ --form 'ApplicationTypeId="13"' \ --form 'Country="ITA"' \ --form 'DocumentType="@ID_CARD"' \ --form 'BacksideImage=@"/C:/Users/n.aleksic/Desktop/BTT flow .xlsx"' \ --form 'ApplicationTypeId="11"' DocumentType parameter moved to Files upload. Only one HTTP_X_FORWARDED_FOR header and ApplicationTypeId, BacksideImage parameters are imported. - And definitely the bug: 3. If URL, header or parameter value is blank, cURL is not imported. cURL example - URL: curl --location --request POST '' \ --header 'Accept-Language: it-IT' \ --header 'HTTP_X_FORWARDED_FOR: 127.0.0.1' \ --form 'FrontsideImage=@"/C:/Users/someuser/Desktop/Desktop.avi"' \ --form 'BacksideImage=@"/C:/Users/someuser/Desktop/Desktop.jpg"' \ --form 'ApplicationTypeId="13"' \ --form 'Country="ITA"' \ --form 'DocumentType="ID_CARD"' Error: ERROR o.a.j.p.h.g.a.ParseCurlCommandAction: Error creating test plan from cURL command:curl --location --request POST '' \ --header 'Accept-Language: it-IT' \ --header 'HTTP_X_FORWARDED_FOR: 127.0.0.1' \ --form 'FrontsideImage=@"/C:/Users/someuser/Desktop/Desktop.avi"' \ --form 'BacksideImage=@"/C:/Users/someuser/Desktop/Desktop.jpg"' \ --form 'ApplicationTypeId="13"' \ --form 'Country="ITA"' \ --form 'DocumentType="ID_CARD"', error:Unexpected format for command line:curl --location --request POST '' \ --header 'Accept-Language: it-IT' \ --header 'HTTP_X_FORWARDED_FOR: 127.0.0.1' \ --form 'FrontsideImage=@"/C:/Users/someuser/Desktop/Desktop.avi"' \ --form 'BacksideImage=@"/C:/Users/someuser/Desktop/Desktop.jpg"' \ --form 'ApplicationTypeId="13"' \ --form 'Country="ITA"' \ --form 'DocumentType="ID_CARD"', error:Unknown option -h java.lang.IllegalArgumentException: Unexpected format for command line:curl --location --request POST '' \ --header 'Accept-Language: it-IT' \ --header 'HTTP_X_FORWARDED_FOR: 127.0.0.1' \ --form 'FrontsideImage=@"/C:/Users/someuser/Desktop/Desktop.avi"' \ --form 'BacksideImage=@"/C:/Users/someuser/Desktop/Desktop.jpg"' \ --form 'ApplicationTypeId="13"' \ --form 'Country="ITA"' \ --form 'DocumentType="ID_CARD"', error:Unknown option -h at org.apache.jmeter.protocol.http.curl.BasicCurlParser.parse(BasicCurlParser.java:797) ~[ApacheJMeter_http.jar:5.4.1-SNAPSHOT] at org.apache.jmeter.protocol.http.gui.action.ParseCurlCommandAction.parseCommands(ParseCurlCommandAction.java:670) ~[ApacheJMeter_http.jar:5.4.1-SNAPSHOT] at org.apache.jmeter.protocol.http.gui.action.ParseCurlCommandAction.actionPerformed(ParseCurlCommandAction.java:619) ~[ApacheJMeter_http.jar:5.4.1-SNAPSHOT] at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1967) ~[?:?] at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2308) ~[?:?] at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:405) ~[?:?] at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:262) ~[?:?] at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:279) ~[?:?] at java.awt.Component.processMouseEvent(Component.java:6636) ~[?:?] at javax.swing.JComponent.processMouseEvent(JComponent.java:3342) ~[?:?] at java.awt.Component.processEvent(Component.java:6401) ~[?:?] at java.awt.Container.processEvent(Container.java:2263) ~[?:?] at
[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013 Felix Schumacher changed: What|Removed |Added Status|NEW |NEEDINFO --- Comment #3 from Felix Schumacher --- Can you test the next build from trunk or nightly and report back, whether it fixes your issue? commit bb0c4557e10278f334844f8c1af873b708981b08 AuthorDate: Tue Dec 22 11:19:21 2020 +0100 POST multipart/form-data cURL code with quoted arguments is not imported correctly Bugzilla Id: 65013 --- .../jmeter/protocol/http/curl/BasicCurlParser.java | 17 --- .../http/gui/action/ParseCurlCommandAction.java| 9 +- .../apache/jmeter/curl/BasicCurlParserTest.java| 34 ++ .../gui/action/ParseCurlCommandActionTest.java | 23 +++ xdocs/changes.xml | 1 + 5 files changed, 79 insertions(+), 5 deletions(-) -- You are receiving this mail because: You are the assignee for the bug.
[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013 Felix Schumacher changed: What|Removed |Added Keywords||PatchAvailable -- You are receiving this mail because: You are the assignee for the bug.
[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013 --- Comment #2 from Felix Schumacher --- Created attachment 37635 --> https://bz.apache.org/bugzilla/attachment.cgi?id=37635=edit unquote form parameters -- You are receiving this mail because: You are the assignee for the bug.
[Bug 65013] POST multipart/form-data cURL code generated from Postman is not imported correctly
https://bz.apache.org/bugzilla/show_bug.cgi?id=65013 Felix Schumacher changed: What|Removed |Added OS||All --- Comment #1 from Felix Schumacher --- The used curl command from the zip file is: curl --location --request POST 'http://example.com/api/somepath' \ --header 'Accept-Language: it-IT' \ --header 'HTTP_X_FORWARDED_FOR: 127.0.0.1' \ --form 'FrontsideImage=@"/C:/Users/someuser/Desktop/Desktop.jpg"' \ --form 'BacksideImage=@"/C:/Users/someuser/Desktop/Desktop.jpg"' \ --form 'ApplicationTypeId="13"' \ --form 'Country="ITA"' \ --form 'DocumentType="ID_CARD"' The filenames are placed together with their quotes into the fields in JMeter (as shown in the images in the zip). The quotes from around the values of the form parameters are kept, too. (Shown again in the images inside the zip). So there are two problems reported here: 1) quotes around the values are kept, but curl would drop them 2) the mime type is set to a very generel mime-type (probably curl is doing a better job at guessing the mime-type) -- You are receiving this mail because: You are the assignee for the bug.
