[Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded
As suggested, here is a working solution using a Role::Tiny role to
modify/override Mojo::Parameters::to_string so that it does not escape the
caret '^' character.
Feel free to comment/criticize.
Thank you all for your valuable input, this was enlightening.
use Mojo::UserAgent;
use FindBin qw($Bin);
use feature qw(say);
use lib $Bin;
$| = 1;
# Accept responses of indefinite size
my $ua = Mojo::UserAgent->new(max_response_size => 0);
$ua->inactivity_timeout(0);
my $url_raw
= 'https://streamerapi.finance.yahoo.com/streamer/1.0?s=
^GSPC&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US®ion=US&localize=0'
;
# Build a normal transaction
my $tx = $ua->build_tx(GET => $url_raw, => {Accept => '*/*'});
# Bypass caret '^' encoding with a Role::Tiny role as Yahoo finance servers
are not RFC 3986 compliant yet...
$tx->req->url->query->with_roles('+Unescaped');
say $tx->req->to_string;
# Replace "read" events to disable default content parser
$tx->res->content->unsubscribe('read')->on(
read => sub {
my ($content, $bytes) = @_;
say "Streaming: $bytes";
}
);
# Process transaction
$tx = $ua->start($tx);
say "done";
And the file ./Mojo/Parameters/Role/Unescaped.pm to modify/override
Mojo::Parameters::to_string
package Mojo::Parameters::Role::Unescaped;
use Mojo::Base -role;
use Mojo::Util qw(decode encode url_escape url_unescape);
has charset => 'UTF-8';
# Override default Mojo::Parameters::to_string method
sub to_string {
my $self = shift;
# String (RFC 3986)
my $charset = $self->charset;
if (defined(my $str = $self->{string})) {
$str = encode $charset, $str if $charset;
# Do not encode caret '^' character
return url_escape $str, '^A-Za-z0-9\-._~%!$&\'()*+,;=:@/?i^';
}
}
1;
On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote:
>
> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
>
> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
> The server returns NOT FOUND.
>
> So this transaction from Mojo::UserAgent returns NOT FOUND:
>
> GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1
> Host: somehost.com
> Accept: */*
> User-Agent: Mojolicious (Perl)
> Content-Length: 0
>
> This transaction using curl with the caret not encoded returns the desired
> output:
>
> GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1
> Host: s <http://streamerapi.finance.yahoo.com/>omehost.com
> User-Agent: curl/7.61.1
> Accept: */*
>
> How does one send a caret '^' in a URL without encoding it to %5E ?
>
> Using a symbol without a caret works great in the Mojo::UserAgent version
> of the code.
>
> Thanks,
>
> Sylvain Thibault
>
On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote:
>
> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
>
> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
> The server returns NOT FOUND.
>
> So this transaction from Mojo::UserAgent returns NOT FOUND:
>
> GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1
> Host: somehost.com
> Accept: */*
> User-Agent: Mojolicious (Perl)
> Content-Length: 0
>
> This transaction using curl with the caret not encoded returns the desired
> output:
>
> GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1
> Host: s <http://streamerapi.finance.yahoo.com/>omehost.com
> User-Agent: curl/7.61.1
> Accept: */*
>
> How does one send a caret '^' in a URL without encoding it to %5E ?
>
> Using a symbol without a caret works great in the Mojo::UserAgent version
> of the code.
>
> Thanks,
>
> Sylvain Thibault
>
>
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
[Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded
As suggested, here is a working solution using a Role::Tiny role to modify
Mojo::Parameters::to_string so that it does not escape the caret '^'
character.
Feel free to comment/criticize.
Thank you all for you valuable input, this was enlightening.
use Mojo::UserAgent;
use FindBin qw($Bin);
use feature qw(say);
use lib $Bin;
$| = 1;
# Accept responses of indefinite size
my $ua = Mojo::UserAgent->new(max_response_size => 0);
$ua->inactivity_timeout(0);
my $url_raw
=
'https://streamerapi.finance.yahoo.com/streamer/1.0?s=^GSPC&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US®ion=US&localize=0'
;
# Build a normal transaction
my $tx = $ua->build_tx(GET => $url_raw, => {Accept => '*/*'});
# Bypass caret '^' encoding as Yahoo finance servers are not RFC 3986
compliant yet...
$tx->req->url->query->with_roles('+Unescaped');
say $tx->req->to_string;
# Replace "read" events to disable default content parser
$tx->res->content->unsubscribe('read')->on(
read => sub {
my ($content, $bytes) = @_;
say "Streaming: $bytes";
}
);
# Process transaction
$tx = $ua->start($tx);
say "done";
And the file ./Mojo/Parameters/Role/Unescaped.pm to modify/override
Mojo::Parameters::to_string
package Mojo::Parameters::Role::Unescaped;
use Mojo::Base -role;
use Mojo::Util qw(decode encode url_escape url_unescape);
has charset => 'UTF-8';
sub to_string {
my $self = shift;
# String (RFC 3986)
my $charset = $self->charset;
if (defined(my $str = $self->{string})) {
$str = encode $charset, $str if $charset;
return url_escape $str, '^A-Za-z0-9\-._~%!$&\'()*+,;=:@/?i^';
}
}
1;
On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote:
>
> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
>
> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
> The server returns NOT FOUND.
>
> So this transaction from Mojo::UserAgent returns NOT FOUND:
>
> GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1
> Host: somehost.com
> Accept: */*
> User-Agent: Mojolicious (Perl)
> Content-Length: 0
>
> This transaction using curl with the caret not encoded returns the desired
> output:
>
> GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1
> Host: s <http://streamerapi.finance.yahoo.com/>omehost.com
> User-Agent: curl/7.61.1
> Accept: */*
>
> How does one send a caret '^' in a URL without encoding it to %5E ?
>
> Using a symbol without a caret works great in the Mojo::UserAgent version
> of the code.
>
> Thanks,
>
> Sylvain Thibault
>
>
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
[Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded
Can somebody describe with a little more detail how one would go about
using a role to modify Mojo::Parameters::to_string
and using it with something like
$tx->req->url->query->with_roles('+Unescaped') ?
I am looking at Joel Berger's blog
post: https://mojolicious.io/blog/2017/12/13/day-13-more-about-roles/
but not quite getting there.
On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote:
>
> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
>
> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
> The server returns NOT FOUND.
>
> So this transaction from Mojo::UserAgent returns NOT FOUND:
>
> GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1
> Host: somehost.com
> Accept: */*
> User-Agent: Mojolicious (Perl)
> Content-Length: 0
>
> This transaction using curl with the caret not encoded returns the desired
> output:
>
> GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1
> Host: s <http://streamerapi.finance.yahoo.com/>omehost.com
> User-Agent: curl/7.61.1
> Accept: */*
>
> How does one send a caret '^' in a URL without encoding it to %5E ?
>
> Using a symbol without a caret works great in the Mojo::UserAgent version
> of the code.
>
> Thanks,
>
> Sylvain Thibault
>
>
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Re: [Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded
Thank you Heiko,
This works for me.
This case is definitely an outlier.
On Wednesday, October 31, 2018 at 5:09:43 AM UTC-6, Heiko Jansen wrote:
>
> I do not know of a non-ugly way either but if you are okay with a hacky
> and possibly fragile solution you might get away with adding:
>
> $ua->on(start => sub {
>my ($ua, $tx) = @_;
>$tx->req->{'start_buffer'} =~ s/([&?]s=)%5E/$1^/o;
> });
>
> Poking into internals usually means one day you'll get what you asked for,
> but if there is no other way...
>
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
Re: [Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded
Thanks for that.
You would think Yahoo would be RFC 3986 compliant but their services have
been on the decline for a while.
I don't know how to override Mojo::Util url_escape either.
On Sunday, October 28, 2018 at 12:46:36 PM UTC-6, Stefan Adams wrote:
>
> I see, you are building a client app, not a server app -- I missed that
> detail.
>
> And, I see that you want to pass the ^ *without* escaping it.
>
> Your URL is getting parsed by Mojo::URL, whose parameters are getting
> parsed by Mojo::Parameters, and when you get the parameters as a string
> <https://mojolicious.org/perldoc/Mojo/Parameters#to_string>, url_escape
> <https://mojolicious.org/perldoc/Mojo/Util#url_escape> is being applied:
>
> Percent encode unsafe characters in string as described in RFC 3986
> <http://tools.ietf.org/html/rfc3986>, the pattern used defaults to
> ^A-Za-z0-9\-._~.
>
>
> Reading the code for to_string in Mojo::Parameters, I see no way around
> this. It would seem that Yahoo Finance is not RFC 3986 compliant. The
> question that needs to be asked then is, "Is it possible for Mojo::URL to
> customize the characters that are considered unsafe?" I don't know super
> advanced Perl like this, but maybe it's possible to override a function and
> make Mojo::Util::url_escape not actually escape the string?
>
> There might be a more advanced way to build your transaction, one that
> doesn't operate on a Mojo::URL object which will ultimately apply
> url_escape without question -- but I wouldn't know it.
>
> Anyway, from what I can see, there's not a solution to your problem. :(
>
>
> On Sun, Oct 28, 2018 at 11:49 AM Sylvain Thibault > wrote:
>
>> Thank you for your response Stefan.
>> Here is the complete code. Symbol SPY works, symbol ^GSPC returns not
>> found.
>>
>> This curl command with the ^GSPC symbol works:
>>
>> curl -s -o - -N -v 'https://streamerapi.finance.yahoo.com/streamer/1.0?s=
>> ^GSPC&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US®ion=US&localize=0'
>>
>>
>> #! /usr/bin/env perl
>> use Mojo::UserAgent;
>> use feature qw(say);
>>
>> $| = 1;
>>
>> # Accept responses of indefinite size
>> my $ua = Mojo::UserAgent->new(max_response_size => 0);
>> $ua->inactivity_timeout(0);
>>
>> my $url_raw
>> # Symbol ^GSPC returns NOT FOUND
>> # = 'https://streamerapi.finance.yahoo.com/streamer/1.0?s=
>> ^GSPC&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US®ion=US&localize=0';
>> = '
>> https://streamerapi.finance.yahoo.com/streamer/1.0?s=SPY&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US®ion=US&localize=0
>> ';
>>
>>
>> # Build a normal transaction
>> my $tx = $ua->build_tx(
>> GET => $url_raw,
>> => {Accept => '*/*'}
>> );
>>
>> # Remove caret encoding... results is not found... ???
>> say $tx->req->to_string;
>>
>> # Replace "read" events to disable default content parser
>> $tx->res->content->unsubscribe('read')->on(
>> read => sub {
>> my ($content, $bytes) = @_;
>> say "Streaming: $bytes";
>> }
>> );
>>
>> # Process transaction
>> $tx = $ua->start($tx);
>>
>>
>> say "done";
>>
>>
>>
>> On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote:
>>>
>>> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
>>>
>>> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
>>> The server returns NOT FOUND.
>>>
>>> So this transaction from Mojo::UserAgent returns NOT FOUND:
>>>
>>> GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1
>>> Host: somehost.com
>>> Accept: */*
>>> User-Agent: Mojolicious (Perl)
>>> Content-Length: 0
>>>
>>> This transaction using curl with the caret not encoded returns the
>>> desired output:
>>>
>>> GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1
>>> Host: s <http://streamerapi.finance.yahoo.com/>omehost.com
>>> User-Agent: curl/7.61.1
>>> Accept: */*
>>>
>>> How does one send a caret '^' in a URL without encoding it to %5E ?
>>>
[Mojolicious] Re: Mojo::UserAgent URL with caret '^' not encoded
Thank you for your response Stefan.
Here is the complete code. Symbol SPY works, symbol ^GSPC returns not found
.
This curl command with the ^GSPC symbol works:
curl -s -o - -N -v
'https://streamerapi.finance.yahoo.com/streamer/1.0?s=^GSPC&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US®ion=US&localize=0'
#! /usr/bin/env perl
use Mojo::UserAgent;
use feature qw(say);
$| = 1;
# Accept responses of indefinite size
my $ua = Mojo::UserAgent->new(max_response_size => 0);
$ua->inactivity_timeout(0);
my $url_raw
# Symbol ^GSPC returns NOT FOUND
# =
'https://streamerapi.finance.yahoo.com/streamer/1.0?s=^GSPC&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US®ion=US&localize=0';
=
'https://streamerapi.finance.yahoo.com/streamer/1.0?s=SPY&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US®ion=US&localize=0'
;
# Build a normal transaction
my $tx = $ua->build_tx(
GET => $url_raw,
=> {Accept => '*/*'}
);
# Remove caret encoding... results is not found... ???
say $tx->req->to_string;
# Replace "read" events to disable default content parser
$tx->res->content->unsubscribe('read')->on(
read => sub {
my ($content, $bytes) = @_;
say "Streaming: $bytes";
}
);
# Process transaction
$tx = $ua->start($tx);
say "done";
On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote:
>
> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC
>
> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E
> The server returns NOT FOUND.
>
> So this transaction from Mojo::UserAgent returns NOT FOUND:
>
> GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1
> Host: somehost.com
> Accept: */*
> User-Agent: Mojolicious (Perl)
> Content-Length: 0
>
> This transaction using curl with the caret not encoded returns the desired
> output:
>
> GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1
> Host: s <http://streamerapi.finance.yahoo.com/>omehost.com
> User-Agent: curl/7.61.1
> Accept: */*
>
> How does one send a caret '^' in a URL without encoding it to %5E ?
>
> Using a symbol without a caret works great in the Mojo::UserAgent version
> of the code.
>
> Thanks,
>
> Sylvain Thibault
>
>
--
You received this message because you are subscribed to the Google Groups
"Mojolicious" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to mojolicious+unsubscr...@googlegroups.com.
To post to this group, send email to mojolicious@googlegroups.com.
Visit this group at https://groups.google.com/group/mojolicious.
For more options, visit https://groups.google.com/d/optout.
[Mojolicious] Mojo::UserAgent URL with caret '^' not encoded
Given URL https://somehost.com.com/streamer/1.0?s=^GSPC When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E The server returns NOT FOUND. So this transaction from Mojo::UserAgent returns NOT FOUND: GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1 Host: somehost.com Accept: */* User-Agent: Mojolicious (Perl) Content-Length: 0 This transaction using curl with the caret not encoded returns the desired output: GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1 Host: s <http://streamerapi.finance.yahoo.com/>omehost.com User-Agent: curl/7.61.1 Accept: */* How does one send a caret '^' in a URL without encoding it to %5E ? Using a symbol without a caret works great in the Mojo::UserAgent version of the code. Thanks, Sylvain Thibault -- You received this message because you are subscribed to the Google Groups "Mojolicious" group. To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+unsubscr...@googlegroups.com. To post to this group, send email to mojolicious@googlegroups.com. Visit this group at https://groups.google.com/group/mojolicious. For more options, visit https://groups.google.com/d/optout.
[Mojolicious] Mojo::UserAgent query with caret '^'
Given URL https://somehost.com.com/streamer/1.0?s=^GSPC When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E The server returns NOT FOUND. So this transaction from Mojo::UserAgent returns NOT FOUND: GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1 Host: somehost.com Accept: */* User-Agent: Mojolicious (Perl) Content-Length: 0 This transaction using curl with the caret not encoded returns the desired output: GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1 Host: streamerapi.finance.yahoo.com User-Agent: curl/7.61.1 Accept: */* How does one send a caret '^' in a URL without encoding it to %5E ? Using a symbol without a caret works great in the Mojo::UserAgent version of the code. Thanks, Sylvain Thibault -- You received this message because you are subscribed to the Google Groups "Mojolicious" group. To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+unsubscr...@googlegroups.com. To post to this group, send email to mojolicious@googlegroups.com. Visit this group at https://groups.google.com/group/mojolicious. For more options, visit https://groups.google.com/d/optout.
