Re: [Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded
On Wed, Oct 31, 2018 at 9:34 PM Sylvain Thibault wrote: > As suggested, here is a working solution using a Role::Tiny role to > modify/override Mojo::Parameters::to_string so that it does not escape the > caret '^' character. > Feel free to comment/criticize. > That looks terrific to me and pretty much exactly what I'd do! > Thank you all for your valuable input, this was enlightening. > Heck yeah! I learned an important lesson myself! Thanks for your question! -- You received this message because you are subscribed to the Google Groups "Mojolicious" group. To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+unsubscr...@googlegroups.com. To post to this group, send email to mojolicious@googlegroups.com. Visit this group at https://groups.google.com/group/mojolicious. For more options, visit https://groups.google.com/d/optout.
[Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded
As suggested, here is a working solution using a Role::Tiny role to
modify/override Mojo::Parameters::to_string so that it does not escape the
caret '^' character.
Feel free to comment/criticize.
Thank you all for your valuable input, this was enlightening.
use Mojo::UserAgent;
use FindBin qw($Bin);
use feature qw(say);
use lib $Bin;
$| = 1;
# Accept responses of indefinite size
my $ua = Mojo::UserAgent->new(max_response_size => 0);
$ua->inactivity_timeout(0);
my $url_raw
= 'https://streamerapi.finance.yahoo.com/streamer/1.0?s=
^GSPC=l86,l84,p20=parent.yfs_u1f=parent.yfs_mktmcb=parent.yfs_gencb=1=en-US=US=0'
;
# Build a normal transaction
my $tx = $ua->build_tx(GET => $url_raw, => {Accept => '*/*'});
# Bypass caret '^' encoding with a Role::Tiny role as Yahoo finance servers
are not RFC 3986 compliant yet...
$tx->req->url->query->with_roles('+Unescaped');
say $tx->req->to_string;
# Replace "read" events to disable default content parser
$tx->res->content->unsubscribe('read')->on(
read => sub {
my ($content, $bytes) = @_;
say "Streaming: $bytes";
}
);
# Process transaction
$tx = $ua->start($tx);
say "done";
And the file ./Mojo/Parameters/Role/Unescaped.pm to modify/override
Mojo::Parameters::to_string
package Mojo::Parameters::Role::Unescaped;
use Mojo::Base -role;
use Mojo::Util qw(decode encode url_escape url_unescape);
has charset => 'UTF-8';
# Override default Mojo::Parameters::to_string method
sub to_string {
my $self = shift;
# String (RFC 3986)
my $charset = $self->charset;
if (defined(my $str = $self->{string})) {
$str = encode $charset, $str if $charset;
# Do not encode caret '^' character
return url_escape $str, '^A-Za-z0-9\-._~%!$&\'()*+,;=:@/?i^';
}
}
1;
On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote:
>
> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
>
> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
> The server returns NOT FOUND.
>
> So this transaction from Mojo::UserAgent returns NOT FOUND:
>
> GET /streamer/1.0?s=%5EGSPC=l86,l84,p20 HTTP/1.1
> Host: somehost.com
> Accept: */*
> User-Agent: Mojolicious (Perl)
> Content-Length: 0
>
> This transaction using curl with the caret not encoded returns the desired
> output:
>
> GET /streamer/1.0?s=^GSPC=l86,l84,p20 HTTP/1.1
> Host: s
[Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded
As suggested, here is a working solution using a Role::Tiny role to modify
Mojo::Parameters::to_string so that it does not escape the caret '^'
character.
Feel free to comment/criticize.
Thank you all for you valuable input, this was enlightening.
use Mojo::UserAgent;
use FindBin qw($Bin);
use feature qw(say);
use lib $Bin;
$| = 1;
# Accept responses of indefinite size
my $ua = Mojo::UserAgent->new(max_response_size => 0);
$ua->inactivity_timeout(0);
my $url_raw
=
'https://streamerapi.finance.yahoo.com/streamer/1.0?s=^GSPC=l86,l84,p20=parent.yfs_u1f=parent.yfs_mktmcb=parent.yfs_gencb=1=en-US=US=0'
;
# Build a normal transaction
my $tx = $ua->build_tx(GET => $url_raw, => {Accept => '*/*'});
# Bypass caret '^' encoding as Yahoo finance servers are not RFC 3986
compliant yet...
$tx->req->url->query->with_roles('+Unescaped');
say $tx->req->to_string;
# Replace "read" events to disable default content parser
$tx->res->content->unsubscribe('read')->on(
read => sub {
my ($content, $bytes) = @_;
say "Streaming: $bytes";
}
);
# Process transaction
$tx = $ua->start($tx);
say "done";
And the file ./Mojo/Parameters/Role/Unescaped.pm to modify/override
Mojo::Parameters::to_string
package Mojo::Parameters::Role::Unescaped;
use Mojo::Base -role;
use Mojo::Util qw(decode encode url_escape url_unescape);
has charset => 'UTF-8';
sub to_string {
my $self = shift;
# String (RFC 3986)
my $charset = $self->charset;
if (defined(my $str = $self->{string})) {
$str = encode $charset, $str if $charset;
return url_escape $str, '^A-Za-z0-9\-._~%!$&\'()*+,;=:@/?i^';
}
}
1;
On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote:
>
> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
>
> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
> The server returns NOT FOUND.
>
> So this transaction from Mojo::UserAgent returns NOT FOUND:
>
> GET /streamer/1.0?s=%5EGSPC=l86,l84,p20 HTTP/1.1
> Host: somehost.com
> Accept: */*
> User-Agent: Mojolicious (Perl)
> Content-Length: 0
>
> This transaction using curl with the caret not encoded returns the desired
> output:
>
> GET /streamer/1.0?s=^GSPC=l86,l84,p20 HTTP/1.1
> Host: s
[Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded
Can somebody describe with a little more detail how one would go about
using a role to modify Mojo::Parameters::to_string
and using it with something like
$tx->req->url->query->with_roles('+Unescaped') ?
I am looking at Joel Berger's blog
post: https://mojolicious.io/blog/2017/12/13/day-13-more-about-roles/
but not quite getting there.
On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote:
>
> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
>
> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
> The server returns NOT FOUND.
>
> So this transaction from Mojo::UserAgent returns NOT FOUND:
>
> GET /streamer/1.0?s=%5EGSPC=l86,l84,p20 HTTP/1.1
> Host: somehost.com
> Accept: */*
> User-Agent: Mojolicious (Perl)
> Content-Length: 0
>
> This transaction using curl with the caret not encoded returns the desired
> output:
>
> GET /streamer/1.0?s=^GSPC=l86,l84,p20 HTTP/1.1
> Host: s
Re: [Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded
On Wed, Oct 31, 2018 at 12:19 PM Dan Book wrote: > Mojo::Collection and Mojo::File are not hash-based objects so they cannot > use Mojo::Base, thus they implement with_roles themselves. Mojo::Base based > objects all say "inherits all methods from Mojo::Base" or a subclass of > that. > This is really helpful! Thank you for clearing that up! -- You received this message because you are subscribed to the Google Groups "Mojolicious" group. To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+unsubscr...@googlegroups.com. To post to this group, send email to mojolicious@googlegroups.com. Visit this group at https://groups.google.com/group/mojolicious. For more options, visit https://groups.google.com/d/optout.
Re: [Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded
On Wed, Oct 31, 2018 at 10:48 AM sri wrote:
> Poking into internals usually means one day you'll get what you asked for,
>> but if there is no other way...
>>
>
> Yes, that is a very dangerous hack, a more sane approach would be to use a
> Role::Tiny role modifying
> Mojo::Parameters::to_string
> ($tx->req->url->query->with_roles('+Unescaped')) or something similar.
>
Yes! I love this strategy! Existing packages like Mojo::Collection and
Mojo::File already have with_roles available, but Mojo::Parameters does
not. Does it make sense to add that, or is it really that simple to just
add it independently and the reason why with_roles is available to these
other packages is just for convenience due to the overwhelming likelihood
that they will be used more frequently?
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Re: [Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded
>
> I do not know of a non-ugly way either but if you are okay with a hacky
> and possibly fragile solution you might get away with adding:
>
> $ua->on(start => sub {
>my ($ua, $tx) = @_;
>$tx->req->{'start_buffer'} =~ s/([&?]s=)%5E/$1^/o;
> });
>
> Poking into internals usually means one day you'll get what you asked for,
> but if there is no other way...
>
Yes, that is a very dangerous hack, a more sane approach would be to use a
Role::Tiny role modifying
Mojo::Parameters::to_string
($tx->req->url->query->with_roles('+Unescaped')) or something similar.
--
sebastian
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Re: [Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded
Thank you Heiko,
This works for me.
This case is definitely an outlier.
On Wednesday, October 31, 2018 at 5:09:43 AM UTC-6, Heiko Jansen wrote:
>
> I do not know of a non-ugly way either but if you are okay with a hacky
> and possibly fragile solution you might get away with adding:
>
> $ua->on(start => sub {
>my ($ua, $tx) = @_;
>$tx->req->{'start_buffer'} =~ s/([&?]s=)%5E/$1^/o;
> });
>
> Poking into internals usually means one day you'll get what you asked for,
> but if there is no other way...
>
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Re: [Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded
I do not know of a non-ugly way either but if you are okay with a hacky and
possibly fragile solution you might get away with adding:
$ua->on(start => sub {
my ($ua, $tx) = @_;
$tx->req->{'start_buffer'} =~ s/([&?]s=)%5E/$1^/o;
});
Poking into internals usually means one day you'll get what you asked for,
but if there is no other way...
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
