Re: [Mojolicious] Re: curl works, Mojo::UserAgent doesn't

[Dan Book]

The built in "multipart" generator is pretty similar to this. You need to
create the parts and set the content type yourself still but it does give
you a little flexibility with hashrefs. See the end of the docs for
https://metacpan.org/pod/Mojo::UserAgent::Transactor#tx

-Dan

On Fri, May 15, 2020 at 12:10 AM Stefan Adams  wrote:

>
>
> On Thu, May 14, 2020 at 7:52 AM 'Michael Lackhoff' via Mojolicious <
> mojolicious@googlegroups.com> wrote:
>
>> When I change the sequence of the form parameters in curl, it fails in
>> the same way. So I guess the sequence is important and my question now is:
>>   Is there a way to pass the form parameters with fixed sequence in M::U
>> (the form-hash of course has a random order)? Perhaps as an array?
>>
>
> Good question.  Probably what you want then is to override the built-in "
> form "
> generator.  You can either override (by simply redefining it), or you can 
> create
> your own generator
> 
> and use it.
>
> $ua->transactor->add_generator(fritz => sub {
>   my ($t, $tx, $args) = @_;
>   my ($sid, $pwd, $config) = @$args;
>   my $sorted_parts = [...];
>   $tx->req->content->parts($sorted_parts);
> });
> $ua->post('http://fritz.box/cgi-bin/firmwarecfg' => fritz => [$SID,
> $BAKPWD, undef]);
>
> You could use a hash instead of an array for your fritz args if you wanted
> to, and still pull out the pieces you want to assemble the array as
> desired.  It's very flexible.
>
> But since you're currently using the "form" generator and learning a good
> deal about it and have determined a solution using the form generator, you
> might want to copy the form generator into your own fritz generator and
> then do the differences you need to (as you wished).  Or you might be able
> to simplify it and just do the content->parts as you alluded to.
> TMTOWTDI!  :D
>
> --
> You received this message because you are subscribed to the Google Groups
> "Mojolicious" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mojolicious+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mojolicious/CACyQ%2BFRSawHXv92ScBL8nc2U_dWVFWwD5KnLqwjCvtKdX8eaPg%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/mojolicious/CABMkAVWK%3Dki20sC_fhzQpWskD9qrB0qygZuJWMTp2Zsy1zewdw%40mail.gmail.com.

Re: [Mojolicious] Re: curl works, Mojo::UserAgent doesn't

[Stefan Adams]

On Thu, May 14, 2020 at 7:52 AM 'Michael Lackhoff' via Mojolicious <
mojolicious@googlegroups.com> wrote:

> When I change the sequence of the form parameters in curl, it fails in the
> same way. So I guess the sequence is important and my question now is:
>   Is there a way to pass the form parameters with fixed sequence in M::U
> (the form-hash of course has a random order)? Perhaps as an array?
>

Good question.  Probably what you want then is to override the built-in "
form "
generator.  You can either override (by simply redefining it), or you
can create
your own generator

and use it.

$ua->transactor->add_generator(fritz => sub {
  my ($t, $tx, $args) = @_;
  my ($sid, $pwd, $config) = @$args;
  my $sorted_parts = [...];
  $tx->req->content->parts($sorted_parts);
});
$ua->post('http://fritz.box/cgi-bin/firmwarecfg' => fritz => [$SID,
$BAKPWD, undef]);

You could use a hash instead of an array for your fritz args if you wanted
to, and still pull out the pieces you want to assemble the array as
desired.  It's very flexible.

But since you're currently using the "form" generator and learning a good
deal about it and have determined a solution using the form generator, you
might want to copy the form generator into your own fritz generator and
then do the differences you need to (as you wished).  Or you might be able
to simplify it and just do the content->parts as you alluded to.
TMTOWTDI!  :D

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/mojolicious/CACyQ%2BFRSawHXv92ScBL8nc2U_dWVFWwD5KnLqwjCvtKdX8eaPg%40mail.gmail.com.

[Mojolicious] Re: curl works, Mojo::UserAgent doesn't

['Michael Lackhoff' via Mojolicious]

Ok, got it :-

1. get the content-parts:
$tx->req->content->parts;
2. by checking each parts body_contains method identify them
3. build a new parts array with correct sequence by copying the objects 
into a new array.
4. change the whole thing:
$tx->req->content->parts($sorted_parts);

This request eventually saves the backup as I want it :-)

You might ask if it is really worth it but I learned quite a lot debugging 
this problem
And there is a wish: please allow adding form parameters as an array. After 
all they end up in an array eventually (parts is an array_ref). It is a 
pitty that the sequence is just determined by a sort on the names i.e. 
arbitrarily.
In most cases it doesn't matter but if it does this saves a lot of fuzz.

-Michael

Am Mittwoch, 13. Mai 2020 23:56:26 UTC+2 schrieb Michael Lackhoff:
>
> I had quite a bit of success recently using Mojo::UserAgent so I tried to 
> replace a curl command to do a backup of my Fritz.box router with M::U.
>
> Here is the curl command:
> curl -s -k -o $OUT --form sid=$SID --form ImportExportPassword=$BAKPWD \
> --form ConfigExport= http://fritz.box/cgi-bin/firmwarecfg
>
> It should be equivalent to this M::U request:
>
> my $tx = $ua->build_tx(
> POST => 'http://fritz.box/cgi-bin/firmwarecfg' =>
> {
> 'Accept'   => '*/*',
> 'Content-Type' => 'multipart/form-data',
> } => form => {
> sid  => $SID,
> ImportExportPassword => $BAKPWD,
> ConfigExport => '',
> }
> );
>
> # for debugging:
> print $tx->req->to_string;
>
> $tx = $ua->start($tx);
> $tx->res->save_to($OUT);
>
> As far as I can tell both the headers and the POST body is very much the 
> same (except the boundary value to separate the form fields) but to my 
> surprise the curl command works ($OUT is the backup file) but with the M::U 
> version $OUT consists of some HTML output indicating an error.
>
> If I could see a difference I could try to better adjust my script but as 
> I said, they look very much the same (I compared it with the -v and 
> --trace-ascii output of curl), so I run out of ideas what could trigger the 
> differnt response of my Fritz.box.
> Any ideas? At the moment I just solve it by using the curl command with 
> "system" but I would prefer a Perl-only solution and what is even more 
> important to me: I want to understand what is going on here.
>
> -Michael
>

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/mojolicious/99009410-9157-443e-ad36-879418436310%40googlegroups.com.

Re: [Mojolicious] Re: curl works, Mojo::UserAgent doesn't

['Michael Lackhoff' via Mojolicious]

I think I am getting closer.
When I change the sequence of the form parameters in curl, it fails in the 
same way. So I guess the sequence is important and my question now is:
  Is there a way to pass the form parameters with fixed sequence in M::U 
(the form-hash of course has a random order)? Perhaps as an array?

-Michael

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/mojolicious/64deb6a2-dfad-4d64-869f-87c2a09f4079%40googlegroups.com.

Re: [Mojolicious] Re: curl works, Mojo::UserAgent doesn't

['Michael Lackhoff' via Mojolicious]

Am Donnerstag, 14. Mai 2020 14:23:41 UTC+2 schrieb Felipe Gasper:
 

> The idea is more that, if you know curl is doing what you want, then just 
> use curl rather than M::UA, and keep on trucking with all else that Mojo 
> has to offer. 
>

Of course you are right. Your suggestion would be the most sensible way of 
dealing with it.
I just happen to have a little daemon sitting on my shoulder, saying "you 
have to understand what is going on here!" "Don't give up until your tool 
that is made for this kind of task does what you want"
It isn't easy for me to silence this little daemon ;-) and so I still hope 
there are others more knowledgable than me in this group who have the same 
kind of ambition...

-Michael

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/mojolicious/7cba2d2e-78bc-43c4-8d36-864ef8991179%40googlegroups.com.

Re: [Mojolicious] Re: curl works, Mojo::UserAgent doesn't

[Felipe Gasper]

Net::Curl::Promiser wouldn’t keep you from using any of Mojo’s other 
capabilities. It’s just an alternative way of doing asynchronous HTTP requests; 
i.e., it’s a substitute specifically for Mojo::UserAgent. It can also 
facilitate use of FTP, IMAP, etc. by virtue of libcurl’s support for those 
protocols.

I wouldn’t necessarily describe the interface itself as “easy” since it’s 
basically a direct port of libcurl’s own interface. The idea is more that, if 
you know curl is doing what you want, then just use curl rather than M::UA, and 
keep on trucking with all else that Mojo has to offer.

-FG

> On May 14, 2020, at 8:06 AM, 'Michael Lackhoff' via Mojolicious 
>  wrote:
> 
> Well, the easiest solution would be to just use curl.
> But M::U is a powerful tool I find myself using it more and more and I would 
> like to better understand such tools.
> It looks great but if it has some severe shortcomings I want to know before 
> doing lots of mission critical stuff with it.
> 
> But thanks for your suggestion. It is always good to know further options.
> -Michael
> 
> 
> Am Donnerstag, 14. Mai 2020 13:47:41 UTC+2 schrieb Felipe Gasper:
> FWIW, you can use libcurl with Mojo via Net::Curl::Promiser::Mojo. That may 
> be your easiest solution?
> 
> -FG
> 
>> On May 14, 2020, at 07:38, 'Michael Lackhoff' via Mojolicious 
>>  wrote:
>> 
>> 
>> Am 14.05.2020 um 05:08 schrieb Stefan Adams:
>> 
>>> > 
>>> I'm certain you'll be able to accomplish this with M::UA. Can you share
>>> 
>> Just to reduce complexity I changed everything to plain HTTP, so no SSL,
>> keys or certificates.
>> 
>> 
>>> > 
>>> what the HTML error produced is?
>>> 
>> It is not really a normal error page, just a page from the UI. The most
>> relevant lines are these:
>> Invalid variable name.
>> Wiederholen Sie das Update oder starten Sie die FRITZ!Box neu.
>> [German for: repeat the update(sic! I wanted to backup the box, not
>> update) or restart your FRITZ!Box]
>> 
>> 
>>> > 
>>> Can you share the output that you are comparing that you are noticing is so
>>> 
>>> > 
>>> similar between curl and M::UA? You're using -v for curl. Also set the
>>> 
>>> > 
>>> MOJO_CLIENT_DEBUG env variable to 1.
>>> 
>> Here is what curl gives (with --trace-ascii):
>> => Send header, 200 bytes (0xc8)
>> : POST /cgi-bin/firmwarecfg HTTP/1.1
>> 0024: Host: fritz.box
>> 0035: User-Agent: curl/7.58.0
>> 004e: Accept: */*
>> 005b: Content-Length: 370
>> 0070: Content-Type: multipart/form-data; boundary=
>> 00b0: 565a580ebeb0c567
>> 00c6:
>> => Send data, 370 bytes (0x172)
>> : --
>> 565a580ebeb0c567
>> 002c: Content-Disposition: form-data; name="sid"
>> 0058:
>> 005a: 4e240384c30c909d
>> 006c: --
>> 565a580ebeb0c567
>> 0098: Content-Disposition: form-data; name="ImportExportPassword"
>> 00d5:
>> 00d7: backup
>> 00df: --
>> 565a580ebeb0c567
>> 010b: Content-Disposition: form-data; name="ConfigExport"
>> 0140:
>> 0142:
>> 0144: --
>> 565a580ebeb0c567--
>> <= Recv header, 17 bytes (0x11)
>> : HTTP/1.1 200 OK
>> 
>> 
>> 
>>> > 
>>> See how it works comparing curl to:
>>> 
>>> > 
>>> > 
>>>$ env MOJO_CLIENT_DEBUG=1 mojo get -v -M POST -f sid=$SID -f
>>> 
>>> > 
>>> ImportExportPassword=$BAKPWD -f ConfigExport= http
>>> 
>>> > ://fritz.box/cgi-bin/firmwarecfg 
>> The empty ConfigExport parameter gets lost with this command and the
>> Content-Type is different:
>> 
>> -- Blocking request (
>> http://fritz.box/cgi-bin/firmwarecfg
>> )
>> -- Connect 7c84d2d7396d9d2ff9571982f73240
>> cb (http://fritz.box:80
>> )
>> -- Client >>> Server (
>> http://fritz.box/cgi-bin/firmwarecfg
>> )
>> POST /cgi-bin/firmwarecfg HTTP/1.1\x0d
>> Content-Length: 48\x0d
>> User-Agent: Mojolicious (Perl)\x0d
>> Accept-Encoding: gzip\x0d
>> Content-Type: application/x-www-form-
>> urlencoded\x0d
>> Host: fritz.box\x0d
>> \x0d
>> ImportExportPassword=backup&
>> sid=4e240384c30c909d
>> -- Client >>> Server (
>> http://fritz.box/cgi-bin/firmwarecfg
>> )
>> 
>> -- Client <<< Server (
>> http://fritz.box/cgi-bin/firmwarecfg
>> )
>> HTTP/1.1 200 OK\x0d
>> 
>> 
>> 
>> With DEBUG on and my code (s. my original message and below) I get this:
>> 
>> -- Client >>> Server (
>> http://fritz.box/cgi-bin/firmwarecfg
>> )
>> POST /cgi-bin/firmwarecfg HTTP/1.1\x0d
>> Accept-Encoding: gzip, deflate\x0d
>> Content-Type: multipart/form-data; boundary=tyUPX\x0d
>> Accept: */*\x0d
>> Host: fritz.box\x0d
>> Content-Length: 230\x0d
>> User-Agent: Mojolicious (Perl)\x0d
>> \x0d
>> --tyUPX\x0d
>> 
>> -- Client >>> Server (
>> http://fritz.box/cgi-bin/firmwarecfg
>> )
>> Content-Disposition: form-data; name="ConfigExport"\x0d
>> \x0d
>> 
>> -- Client >>> Server (
>> http://fritz.box/cgi-bin/firmwarecfg
>> )
>> \x0d
>> --tyUPX\x0d
>> 
>> -- Client >>> Server (
>> http://fritz.box/cgi-bin/firmwarecfg
>> )
>> Content-Disposition: form-data; name="ImportExpo

Re: [Mojolicious] Re: curl works, Mojo::UserAgent doesn't

['Tobias Oetiker' via Mojolicious]

Michael, 

I find the descriptions in [ 
https://mojolicious.org/perldoc/Mojo/UserAgent/Transactor#tx | 
https://mojolicious.org/perldoc/Mojo/UserAgent/Transactor#tx ] pretty helpful 

cheers 
tobi 

- On May 14, 2020, at 2:06 PM, 'Michael Lackhoff' via Mojolicious 
 wrote: 

> Well, the easiest solution would be to just use curl.
> But M::U is a powerful tool I find myself using it more and more and I would
> like to better understand such tools.
> It looks great but if it has some severe shortcomings I want to know before
> doing lots of mission critical stuff with it.

> But thanks for your suggestion. It is always good to know further options.
> -Michael

> Am Donnerstag, 14. Mai 2020 13:47:41 UTC+2 schrieb Felipe Gasper:
>> FWIW, you can use libcurl with Mojo via Net::Curl::Promiser::Mojo. That may 
>> be
>> your easiest solution?

>> -FG

>>> On May 14, 2020, at 07:38, 'Michael Lackhoff' via Mojolicious <
>>> mojol...@googlegroups.com > wrote:

>>> Am 14.05.2020 um 05:08 schrieb Stefan Adams:

 > I'm certain you'll be able to accomplish this with M::UA. Can you share

>>> Just to reduce complexity I changed everything to plain HTTP, so no SSL,
>>> keys or certificates.

 > what the HTML error produced is?

>>> It is not really a normal error page, just a page from the UI. The most
>>> relevant lines are these:
>>> Invalid variable name.
>>> Wiederholen Sie das Update oder starten Sie die FRITZ!Box neu.
>>> [German for: repeat the update(sic! I wanted to backup the box, not
>>> update) or restart your FRITZ!Box]

 > Can you share the output that you are comparing that you are noticing is 
 > so
 > similar between curl and M::UA? You're using -v for curl. Also set the
 > MOJO_CLIENT_DEBUG env variable to 1.

>>> Here is what curl gives (with --trace-ascii):
>>> => Send header, 200 bytes (0xc8)
>>> : POST /cgi-bin/firmwarecfg HTTP/1.1
>>> 0024: Host: fritz.box
>>> 0035: User-Agent: curl/7.58.0
>>> 004e: Accept: */*
>>> 005b: Content-Length: 370
>>> 0070: Content-Type: multipart/form-data; boundary=
>>> 00b0: 565a580ebeb0c567
>>> 00c6:
>>> => Send data, 370 bytes (0x172)
>>> : --565a580ebeb0c567
>>> 002c: Content-Disposition: form-data; name="sid"
>>> 0058:
>>> 005a: 4e240384c30c909d
>>> 006c: --565a580ebeb0c567
>>> 0098: Content-Disposition: form-data; name="ImportExportPassword"
>>> 00d5:
>>> 00d7: backup
>>> 00df: --565a580ebeb0c567
>>> 010b: Content-Disposition: form-data; name="ConfigExport"
>>> 0140:
>>> 0142:
>>> 0144: --565a580ebeb0c567--
>>> <= Recv header, 17 bytes (0x11)
>>> : HTTP/1.1 200 OK

 > See how it works comparing curl to:

 >$ env MOJO_CLIENT_DEBUG=1 mojo get -v -M POST -f sid=$SID -f
 > ImportExportPassword=$BAKPWD -f ConfigExport= http
> ://fritz.box/cgi-bin/firmwarecfg [ http://fritz.box/cgi-bin/firmwarecfg |
 >  ]

>>> The empty ConfigExport parameter gets lost with this command and the
>>> Content-Type is different:

>>> -- Blocking request ( [ http://fritz.box/cgi-bin/firmwarecfg |
>>> http://fritz.box/cgi-bin/firmwarecfg ] )
>>> -- Connect 7c84d2d7396d9d2ff9571982f73240cb ( [ http://fritz.box/ |
>>> http://fritz.box:80 ] )
>>> -- Client >>> Server ( [ http://fritz.box/cgi-bin/firmwarecfg |
>>> http://fritz.box/cgi-bin/firmwarecfg ] )
>>> POST /cgi-bin/firmwarecfg HTTP/1.1\x0d
>>> Content-Length: 48\x0d
>>> User-Agent: Mojolicious (Perl)\x0d
>>> Accept-Encoding: gzip\x0d
>>> Content-Type: application/x-www-form-urlencoded\x0d
>>> Host: fritz.box\x0d
>>> \x0d
>>> ImportExportPassword=backup&sid=4e240384c30c909d
>>> -- Client >>> Server ( [ http://fritz.box/cgi-bin/firmwarecfg |
>>> http://fritz.box/cgi-bin/firmwarecfg ] )

>>> -- Client <<< Server ( [ http://fritz.box/cgi-bin/firmwarecfg |
>>> http://fritz.box/cgi-bin/firmwarecfg ] )
>>> HTTP/1.1 200 OK\x0d

>>> With DEBUG on and my code (s. my original message and below) I get this:

>>> -- Client >>> Server ( [ http://fritz.box/cgi-bin/firmwarecfg |
>>> http://fritz.box/cgi-bin/firmwarecfg ] )
>>> POST /cgi-bin/firmwarecfg HTTP/1.1\x0d
>>> Accept-Encoding: gzip, deflate\x0d
>>> Content-Type: multipart/form-data; boundary=tyUPX\x0d
>>> Accept: */*\x0d
>>> Host: fritz.box\x0d
>>> Content-Length: 230\x0d
>>> User-Agent: Mojolicious (Perl)\x0d
>>> \x0d
>>> --tyUPX\x0d

>>> -- Client >>> Server ( [ http://fritz.box/cgi-bin/firmwarecfg |
>>> http://fritz.box/cgi-bin/firmwarecfg ] )
>>> Content-Disposition: form-data; name="ConfigExport"\x0d
>>> \x0d

>>> -- Client >>> Server ( [ http://fritz.box/cgi-bin/firmwarecfg |
>>> http://fritz.box/cgi-bin/firmwarecfg ] )
>>> \x0d
>>> --tyUPX\x0d

>>> -- Client >>> Server ( [ http://fritz.box/cgi-bin/firmwarecfg |
>>> http://fritz.box/cgi-bin/firmwarecfg ] )
>>> Content-Disposition: form-data; name="ImportExportPassword"\x0d
>>> \x0d

>>> -- Client >>> Server ( [ http://fritz

Re: [Mojolicious] Re: curl works, Mojo::UserAgent doesn't

['Michael Lackhoff' via Mojolicious]

Well, the easiest solution would be to just use curl.
But M::U is a powerful tool I find myself using it more and more and I 
would like to better understand such tools.
It looks great but if it has some severe shortcomings I want to know before 
doing lots of mission critical stuff with it.

But thanks for your suggestion. It is always good to know further options.
-Michael


Am Donnerstag, 14. Mai 2020 13:47:41 UTC+2 schrieb Felipe Gasper:
>
> FWIW, you can use libcurl with Mojo via Net::Curl::Promiser::Mojo. That 
> may be your easiest solution?
>
> -FG
>
> On May 14, 2020, at 07:38, 'Michael Lackhoff' via Mojolicious <
> mojol...@googlegroups.com > wrote:
>
> 
>
> Am 14.05.2020 um 05:08 schrieb Stefan Adams:
>
> > I'm certain you'll be able to accomplish this with M::UA. Can you share
>
> Just to reduce complexity I changed everything to plain HTTP, so no SSL,
> keys or certificates.
>
>
> > what the HTML error produced is?
>
> It is not really a normal error page, just a page from the UI. The most
> relevant lines are these:
> Invalid variable name.
> Wiederholen Sie das Update oder starten Sie die FRITZ!Box neu.
> [German for: repeat the update(sic! I wanted to backup the box, not
> update) or restart your FRITZ!Box]
>
>
> > Can you share the output that you are comparing that you are noticing is 
> > so> similar between curl and M::UA? You're using -v for curl. Also set the> 
> > MOJO_CLIENT_DEBUG env variable to 1.
>
> Here is what curl gives (with --trace-ascii):
> => Send header, 200 bytes (0xc8)
> : POST /cgi-bin/firmwarecfg HTTP/1.1
> 0024: Host: fritz.box
> 0035: User-Agent: curl/7.58.0
> 004e: Accept: */*
> 005b: Content-Length: 370
> 0070: Content-Type: multipart/form-data; boundary=
> 00b0: 565a580ebeb0c567
> 00c6:
> => Send data, 370 bytes (0x172)
> : --565a580ebeb0c567
> 002c: Content-Disposition: form-data; name="sid"
> 0058:
> 005a: 4e240384c30c909d
> 006c: --565a580ebeb0c567
> 0098: Content-Disposition: form-data; name="ImportExportPassword"
> 00d5:
> 00d7: backup
> 00df: --565a580ebeb0c567
> 010b: Content-Disposition: form-data; name="ConfigExport"
> 0140:
> 0142:
> 0144: --565a580ebeb0c567--
> <= Recv header, 17 bytes (0x11)
> : HTTP/1.1 200 OK
>
>
>
> > See how it works comparing curl to:> >$ env MOJO_CLIENT_DEBUG=1 mojo 
> > get -v -M POST -f sid=$SID -f> ImportExportPassword=$BAKPWD -f 
> > ConfigExport= http> ://fritz.box/cgi-bin/firmwarecfg 
> >  
> > 
>
> The empty ConfigExport parameter gets lost with this command and the
> Content-Type is different:
>
> -- Blocking request (http://fritz.box/cgi-bin/firmwarecfg)
> -- Connect 7c84d2d7396d9d2ff9571982f73240cb (http://fritz.box:80)
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> POST /cgi-bin/firmwarecfg HTTP/1.1\x0d
> Content-Length: 48\x0d
> User-Agent: Mojolicious (Perl)\x0d
> Accept-Encoding: gzip\x0d
> Content-Type: application/x-www-form-urlencoded\x0d
> Host: fritz.box\x0d
> \x0d
> ImportExportPassword=backup&sid=4e240384c30c909d
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
>
> -- Client <<< Server (http://fritz.box/cgi-bin/firmwarecfg)
> HTTP/1.1 200 OK\x0d
>
>
>
> With DEBUG on and my code (s. my original message and below) I get this:
>
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> POST /cgi-bin/firmwarecfg HTTP/1.1\x0d
> Accept-Encoding: gzip, deflate\x0d
> Content-Type: multipart/form-data; boundary=tyUPX\x0d
> Accept: */*\x0d
> Host: fritz.box\x0d
> Content-Length: 230\x0d
> User-Agent: Mojolicious (Perl)\x0d
> \x0d
> --tyUPX\x0d
>
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> Content-Disposition: form-data; name="ConfigExport"\x0d
> \x0d
>
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> \x0d
> --tyUPX\x0d
>
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> Content-Disposition: form-data; name="ImportExportPassword"\x0d
> \x0d
>
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> backup
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> \x0d
> --tyUPX\x0d
>
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> Content-Disposition: form-data; name="sid"\x0d
> \x0d
>
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> 33cb272bcd34940b
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> \x0d
> --tyUPX--\x0d
>
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
>
> -- Client <<< Server (http://fritz.box/cgi-bin/firmwarecfg)
> HTTP/1.1 200 OK\x0d
>
> which for me looks very much the same as the curl version but as I said,
> curl gives the backup file whereas M::U just gives the HTML page back,
> indicating that something went wrong.
> I can only spot very minor differences that is why I am running out of
> ideas:
> - the boundary used by M::U is qu

Re: [Mojolicious] Re: curl works, Mojo::UserAgent doesn't

[Felipe Gasper]

FWIW, you can use libcurl with Mojo via Net::Curl::Promiser::Mojo. That may be 
your easiest solution?

-FG

> On May 14, 2020, at 07:38, 'Michael Lackhoff' via Mojolicious 
>  wrote:
> 
> 
>> Am 14.05.2020 um 05:08 schrieb Stefan Adams:
>> > I'm certain you'll be able to accomplish this with M::UA. Can you share
> Just to reduce complexity I changed everything to plain HTTP, so no SSL,
> keys or certificates.
> 
>> > what the HTML error produced is?
> It is not really a normal error page, just a page from the UI. The most
> relevant lines are these:
> Invalid variable name.
> Wiederholen Sie das Update oder starten Sie die FRITZ!Box neu.
> [German for: repeat the update(sic! I wanted to backup the box, not
> update) or restart your FRITZ!Box]
> 
>> > Can you share the output that you are comparing that you are noticing is so
>> > similar between curl and M::UA? You're using -v for curl. Also set the
>> > MOJO_CLIENT_DEBUG env variable to 1.
> Here is what curl gives (with --trace-ascii):
> => Send header, 200 bytes (0xc8)
> : POST /cgi-bin/firmwarecfg HTTP/1.1
> 0024: Host: fritz.box
> 0035: User-Agent: curl/7.58.0
> 004e: Accept: */*
> 005b: Content-Length: 370
> 0070: Content-Type: multipart/form-data; boundary=
> 00b0: 565a580ebeb0c567
> 00c6:
> => Send data, 370 bytes (0x172)
> : --565a580ebeb0c567
> 002c: Content-Disposition: form-data; name="sid"
> 0058:
> 005a: 4e240384c30c909d
> 006c: --565a580ebeb0c567
> 0098: Content-Disposition: form-data; name="ImportExportPassword"
> 00d5:
> 00d7: backup
> 00df: --565a580ebeb0c567
> 010b: Content-Disposition: form-data; name="ConfigExport"
> 0140:
> 0142:
> 0144: --565a580ebeb0c567--
> <= Recv header, 17 bytes (0x11)
> : HTTP/1.1 200 OK
> 
> 
>> > See how it works comparing curl to:
>> > 
>> >$ env MOJO_CLIENT_DEBUG=1 mojo get -v -M POST -f sid=$SID -f
>> > ImportExportPassword=$BAKPWD -f ConfigExport= http
>> > ://fritz.box/cgi-bin/firmwarecfg 
> The empty ConfigExport parameter gets lost with this command and the
> Content-Type is different:
> 
> -- Blocking request (http://fritz.box/cgi-bin/firmwarecfg)
> -- Connect 7c84d2d7396d9d2ff9571982f73240cb (http://fritz.box:80)
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> POST /cgi-bin/firmwarecfg HTTP/1.1\x0d
> Content-Length: 48\x0d
> User-Agent: Mojolicious (Perl)\x0d
> Accept-Encoding: gzip\x0d
> Content-Type: application/x-www-form-urlencoded\x0d
> Host: fritz.box\x0d
> \x0d
> ImportExportPassword=backup&sid=4e240384c30c909d
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> 
> -- Client <<< Server (http://fritz.box/cgi-bin/firmwarecfg)
> HTTP/1.1 200 OK\x0d
> 
> 
> 
> With DEBUG on and my code (s. my original message and below) I get this:
> 
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> POST /cgi-bin/firmwarecfg HTTP/1.1\x0d
> Accept-Encoding: gzip, deflate\x0d
> Content-Type: multipart/form-data; boundary=tyUPX\x0d
> Accept: */*\x0d
> Host: fritz.box\x0d
> Content-Length: 230\x0d
> User-Agent: Mojolicious (Perl)\x0d
> \x0d
> --tyUPX\x0d
> 
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> Content-Disposition: form-data; name="ConfigExport"\x0d
> \x0d
> 
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> \x0d
> --tyUPX\x0d
> 
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> Content-Disposition: form-data; name="ImportExportPassword"\x0d
> \x0d
> 
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> backup
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> \x0d
> --tyUPX\x0d
> 
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> Content-Disposition: form-data; name="sid"\x0d
> \x0d
> 
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> 33cb272bcd34940b
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> \x0d
> --tyUPX--\x0d
> 
> -- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
> 
> -- Client <<< Server (http://fritz.box/cgi-bin/firmwarecfg)
> HTTP/1.1 200 OK\x0d
> 
> which for me looks very much the same as the curl version but as I said,
> curl gives the backup file whereas M::U just gives the HTML page back,
> indicating that something went wrong.
> I can only spot very minor differences that is why I am running out of
> ideas:
> - the boundary used by M::U is quite short
> - the form parameters are not in the same order
> - Accept-Encoding header (is also sent from Firefox where it works)
> - the different SID is normal (my script always makes a fresch login)
> 
> Looks very strange to me
> -Michael
> 
>> > On Wed, May 13, 2020, 4:56 PM 'Michael Lackhoff' via Mojolicious <
>> > mojolicious@googlegroups.com> wrote:
>> > 
>>> >> I had quite a bit of success recently using Mojo::UserAgent so I tried to
>>> >> replace a curl command to do a backup of my Fritz.box router with M:

[Mojolicious] Re: curl works, Mojo::UserAgent doesn't

['Michael Lackhoff' via Mojolicious]

 

Am 14.05.2020 um 05:08 schrieb Stefan Adams:

> I'm certain you'll be able to accomplish this with M::UA. Can you share

Just to reduce complexity I changed everything to plain HTTP, so no SSL,
keys or certificates.


> what the HTML error produced is?

It is not really a normal error page, just a page from the UI. The most
relevant lines are these:
Invalid variable name.
Wiederholen Sie das Update oder starten Sie die FRITZ!Box neu.
[German for: repeat the update(sic! I wanted to backup the box, not
update) or restart your FRITZ!Box]


> Can you share the output that you are comparing that you are noticing is so> 
> similar between curl and M::UA? You're using -v for curl. Also set the> 
> MOJO_CLIENT_DEBUG env variable to 1.

Here is what curl gives (with --trace-ascii):
=> Send header, 200 bytes (0xc8)
: POST /cgi-bin/firmwarecfg HTTP/1.1
0024: Host: fritz.box
0035: User-Agent: curl/7.58.0
004e: Accept: */*
005b: Content-Length: 370
0070: Content-Type: multipart/form-data; boundary=
00b0: 565a580ebeb0c567
00c6:
=> Send data, 370 bytes (0x172)
: --565a580ebeb0c567
002c: Content-Disposition: form-data; name="sid"
0058:
005a: 4e240384c30c909d
006c: --565a580ebeb0c567
0098: Content-Disposition: form-data; name="ImportExportPassword"
00d5:
00d7: backup
00df: --565a580ebeb0c567
010b: Content-Disposition: form-data; name="ConfigExport"
0140:
0142:
0144: --565a580ebeb0c567--
<= Recv header, 17 bytes (0x11)
: HTTP/1.1 200 OK



> See how it works comparing curl to:> >$ env MOJO_CLIENT_DEBUG=1 mojo get 
> -v -M POST -f sid=$SID -f> ImportExportPassword=$BAKPWD -f ConfigExport= 
> http> ://fritz.box/cgi-bin/firmwarecfg  
> 

The empty ConfigExport parameter gets lost with this command and the
Content-Type is different:

-- Blocking request (http://fritz.box/cgi-bin/firmwarecfg)
-- Connect 7c84d2d7396d9d2ff9571982f73240cb (http://fritz.box:80)
-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
POST /cgi-bin/firmwarecfg HTTP/1.1\x0d
Content-Length: 48\x0d
User-Agent: Mojolicious (Perl)\x0d
Accept-Encoding: gzip\x0d
Content-Type: application/x-www-form-urlencoded\x0d
Host: fritz.box\x0d
\x0d
ImportExportPassword=backup&sid=4e240384c30c909d
-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)

-- Client <<< Server (http://fritz.box/cgi-bin/firmwarecfg)
HTTP/1.1 200 OK\x0d



With DEBUG on and my code (s. my original message and below) I get this:

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
POST /cgi-bin/firmwarecfg HTTP/1.1\x0d
Accept-Encoding: gzip, deflate\x0d
Content-Type: multipart/form-data; boundary=tyUPX\x0d
Accept: */*\x0d
Host: fritz.box\x0d
Content-Length: 230\x0d
User-Agent: Mojolicious (Perl)\x0d
\x0d
--tyUPX\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
Content-Disposition: form-data; name="ConfigExport"\x0d
\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
\x0d
--tyUPX\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
Content-Disposition: form-data; name="ImportExportPassword"\x0d
\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
backup
-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
\x0d
--tyUPX\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
Content-Disposition: form-data; name="sid"\x0d
\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
33cb272bcd34940b
-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)
\x0d
--tyUPX--\x0d

-- Client >>> Server (http://fritz.box/cgi-bin/firmwarecfg)

-- Client <<< Server (http://fritz.box/cgi-bin/firmwarecfg)
HTTP/1.1 200 OK\x0d

which for me looks very much the same as the curl version but as I said,
curl gives the backup file whereas M::U just gives the HTML page back,
indicating that something went wrong.
I can only spot very minor differences that is why I am running out of
ideas:
- the boundary used by M::U is quite short
- the form parameters are not in the same order
- Accept-Encoding header (is also sent from Firefox where it works)
- the different SID is normal (my script always makes a fresch login)

Looks very strange to me
-Michael


> On Wed, May 13, 2020, 4:56 PM 'Michael Lackhoff' via Mojolicious <> 
> mojolicious@googlegroups.com> wrote:> 

>> I had quite a bit of success recently using Mojo::UserAgent so I tried to>> 
>> replace a curl command to do a backup of my Fritz.box router with M::U. 
>> Here is the curl command:>> curl -s -k -o $OUT --form sid=$SID --form 
>> ImportExportPassword=$BAKPWD \>> --form ConfigExport= 
>> http://fritz.box/cgi-bin/firmwarecfg It should be equivalent to this 
>> M::U request: my $tx = $ua->build_tx(>> POST => 
>> 'http://fritz.box/cgi-bin/firmwarecfg' =>>> {>> 'Accept' 
>>   => '*/*',>> 'Content-Typ

[Mojolicious] Re: curl works, Mojo::UserAgent doesn't

[Scott Wiersdorf]

It's possible the `-k` option to curl is the difference. The equivalent in 
Mojo::UserAgent is `$ua->insecure(1)` (which you can chain off of).

Scott

On Wednesday, May 13, 2020 at 3:56:26 PM UTC-6, Michael Lackhoff wrote:
>
> I had quite a bit of success recently using Mojo::UserAgent so I tried to 
> replace a curl command to do a backup of my Fritz.box router with M::U.
>
> Here is the curl command:
> curl -s -k -o $OUT --form sid=$SID --form ImportExportPassword=$BAKPWD \
> --form ConfigExport= http://fritz.box/cgi-bin/firmwarecfg
>
> It should be equivalent to this M::U request:
>
> my $tx = $ua->build_tx(
> POST => 'http://fritz.box/cgi-bin/firmwarecfg' =>
> {
> 'Accept'   => '*/*',
> 'Content-Type' => 'multipart/form-data',
> } => form => {
> sid  => $SID,
> ImportExportPassword => $BAKPWD,
> ConfigExport => '',
> }
> );
>
> # for debugging:
> print $tx->req->to_string;
>
> $tx = $ua->start($tx);
> $tx->res->save_to($OUT);
>
> As far as I can tell both the headers and the POST body is very much the 
> same (except the boundary value to separate the form fields) but to my 
> surprise the curl command works ($OUT is the backup file) but with the M::U 
> version $OUT consists of some HTML output indicating an error.
>
> If I could see a difference I could try to better adjust my script but as 
> I said, they look very much the same (I compared it with the -v and 
> --trace-ascii output of curl), so I run out of ideas what could trigger the 
> differnt response of my Fritz.box.
> Any ideas? At the moment I just solve it by using the curl command with 
> "system" but I would prefer a Perl-only solution and what is even more 
> important to me: I want to understand what is going on here.
>
> -Michael
>

-- 
You received this message because you are subscribed to the Google Groups 
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mojolicious+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/mojolicious/61ee3b8a-ca87-424d-91e1-f807001afa37%40googlegroups.com.