Thanks for that. You would think Yahoo would be RFC 3986 compliant but their services have been on the decline for a while.
I don't know how to override Mojo::Util url_escape either. On Sunday, October 28, 2018 at 12:46:36 PM UTC-6, Stefan Adams wrote: > > I see, you are building a client app, not a server app -- I missed that > detail. > > And, I see that you want to pass the ^ *without* escaping it. > > Your URL is getting parsed by Mojo::URL, whose parameters are getting > parsed by Mojo::Parameters, and when you get the parameters as a string > <https://mojolicious.org/perldoc/Mojo/Parameters#to_string>, url_escape > <https://mojolicious.org/perldoc/Mojo/Util#url_escape> is being applied: > > Percent encode unsafe characters in string as described in RFC 3986 > <http://tools.ietf.org/html/rfc3986>, the pattern used defaults to > ^A-Za-z0-9\-._~. > > > Reading the code for to_string in Mojo::Parameters, I see no way around > this. It would seem that Yahoo Finance is not RFC 3986 compliant. The > question that needs to be asked then is, "Is it possible for Mojo::URL to > customize the characters that are considered unsafe?" I don't know super > advanced Perl like this, but maybe it's possible to override a function and > make Mojo::Util::url_escape not actually escape the string? > > There might be a more advanced way to build your transaction, one that > doesn't operate on a Mojo::URL object which will ultimately apply > url_escape without question -- but I wouldn't know it. > > Anyway, from what I can see, there's not a solution to your problem. :( > > > On Sun, Oct 28, 2018 at 11:49 AM Sylvain Thibault <sthi...@smbcap.com > <javascript:>> wrote: > >> Thank you for your response Stefan. >> Here is the complete code. Symbol SPY works, symbol ^GSPC returns not >> found. >> >> This curl command with the ^GSPC symbol works: >> >> curl -s -o - -N -v 'https://streamerapi.finance.yahoo.com/streamer/1.0?s= >> ^GSPC&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US®ion=US&localize=0' >> >> >> #! /usr/bin/env perl >> use Mojo::UserAgent; >> use feature qw(say); >> >> $| = 1; >> >> # Accept responses of indefinite size >> my $ua = Mojo::UserAgent->new(max_response_size => 0); >> $ua->inactivity_timeout(0); >> >> my $url_raw >> # Symbol ^GSPC returns NOT FOUND >> # = 'https://streamerapi.finance.yahoo.com/streamer/1.0?s= >> ^GSPC&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US®ion=US&localize=0'; >> = ' >> https://streamerapi.finance.yahoo.com/streamer/1.0?s=SPY&k=l86,l84,p20&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb&mu=1&lang=en-US®ion=US&localize=0 >> '; >> >> >> # Build a normal transaction >> my $tx = $ua->build_tx( >> GET => $url_raw, >> => {Accept => '*/*'} >> ); >> >> # Remove caret encoding... results is not found... ??? >> say $tx->req->to_string; >> >> # Replace "read" events to disable default content parser >> $tx->res->content->unsubscribe('read')->on( >> read => sub { >> my ($content, $bytes) = @_; >> say "Streaming: $bytes"; >> } >> ); >> >> # Process transaction >> $tx = $ua->start($tx); >> >> >> say "done"; >> >> >> >> On Saturday, October 27, 2018 at 5:38:43 PM UTC-6, Sylvain Thibault wrote: >>> >>> Given URL https://somehost.com.com/streamer/1.0?s=^GSPC >>> >>> When doing a GET, Mojo::UserAgent encodes the caret '^' as %5E >>> The server returns NOT FOUND. >>> >>> So this transaction from Mojo::UserAgent returns NOT FOUND: >>> >>> GET /streamer/1.0?s=%5EGSPC&k=l86,l84,p20 HTTP/1.1 >>> Host: somehost.com >>> Accept: */* >>> User-Agent: Mojolicious (Perl) >>> Content-Length: 0 >>> >>> This transaction using curl with the caret not encoded returns the >>> desired output: >>> >>> GET /streamer/1.0?s=^GSPC&k=l86,l84,p20 HTTP/1.1 >>> Host: s <http://streamerapi.finance.yahoo.com/>omehost.com >>> User-Agent: curl/7.61.1 >>> Accept: */* >>> >>> How does one send a caret '^' in a URL without encoding it to %5E ? >>> >>> Using a symbol without a caret works great in the Mojo::UserAgent >>> version of the code. >>> >>> Thanks, >>> >>> Sylvain Thibault >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Mojolicious" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to mojolicious...@googlegroups.com <javascript:>. >> To post to this group, send email to mojol...@googlegroups.com >> <javascript:>. >> Visit this group at https://groups.google.com/group/mojolicious. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Mojolicious" group. To unsubscribe from this group and stop receiving emails from it, send an email to mojolicious+unsubscr...@googlegroups.com. To post to this group, send email to mojolicious@googlegroups.com. Visit this group at https://groups.google.com/group/mojolicious. For more options, visit https://groups.google.com/d/optout.
