Ric for not looking at my code you are amazing. I put the ‘dquote’ function in front of the URL that goes to get the actual stock quotes and bingo CSV data dumped into my Jqt screen.
It’s the second to last line of my script that was the culprit. It initially was: res2 =: '-s -b cookie.txt ' gethttp URL Now it’s: res2 =: '-s -b cookie.txt ' gethttp dquote URL And the following comes down from yahoo ... 2019-01-10,152.500000,153.970001,150.860001,153.800003,151.523376,35780700 2019-01-11,152.880005,153.699997,151.509995,152.289993,150.035706,27023200 2019-01-14,150.850006,151.270004,149.220001,150.000000,147.779602,32439200 … > On Nov 24, 2019, at 9:55 PM, Ric Sherlock <tikk...@gmail.com> wrote: > > Haven't got time to look at your code in detail, but stabbing blindly in > the dark - have you tried double quoting your URL? > See the following StackOverflow question and answer: > https://stackoverflow.com/questions/55753504/reading-csv-file-from-a-a-secured-link-with-j > > On Mon, Nov 25, 2019 at 3:36 PM Henry Rich <henryhr...@gmail.com> wrote: > >> Do you have an example of a script where you can see correct data going >> in and a bad result coming out? >> >> Henry Rich >> >> On 11/24/2019 9:30 PM, Thomas McGuire wrote: >>> The following J script, patterned after Brad Lucas’ Bash script ( >> http://blog.bradlucas.com/posts/2017-06-02-new-yahoo-finance-quote-download-url/ >> < >> http://blog.bradlucas.com/posts/2017-06-02-new-yahoo-finance-quote-download-url/>), >> doesn’t seem to work. I won’t go into the history of the yahoo and the >> change in their website except to say that by making a double access with >> curl and pulling out elements of the first response to use in the second >> access you should be able to programmatically get historical price quotes. >>> >>> I believe I have set up the URLs properly. I am using cookies with >> web/gethttp function. I see the cookie file get created and I receive a >> first response from the yahoo website. From there you need to find the >> ‘crumb’ in the response and use that to build a URL to request the >> historical stock prices. The crumb can sometimes contain \u00xx unicode >> escape characters. These need to be translated into a character. Bash has a >> built in echo command (different than the linux echo command) that will >> translate this properly. In J I use reapply to translate these sequences. >> The Bash version redirects the output of the second curl call to a file. I >> am just trying to get the data to print in Jqt where I run the script. >>> >>> I will show the output and the J script will follow. I tried this with >> 2!:0 foreign conjunction to run curl directly and have the same problem. >> The first call to curl gets an appropriate response and the second call >> gets an error stating invalid cookie. This error is not displayed in J I >> modified the code to log the errors to a file and found it there. >>> >>> OUTPUT: >>> gethistorical 'AAPL';'11/23/2018';'11/23/2019' >>> start URL = https://finance.yahoo.com/quote/AAPL/history?p=AAPL >>> raw crumb string = >> "}}}}},"CrumbStore":{"crumb":"34jVbQ1iOCI"},"StreamStore":{" >>> final crumb string = 34jVbQ1iOCI >>> >> https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1543010400&period2=1574546400&interval=1d&events=history&crumb=34jVbQ1iOCI >>> >>> OUTPUT (with unicode escape sequence): >>> gethistorical 'AAPL';'11/23/2018';'11/23/2019' >>> start URL = https://finance.yahoo.com/quote/AAPL/?p=AAPL >>> raw crumb string = >> "}}}}},"CrumbStore":{"crumb":"yw3klV8Pa\u002Fc"},"StreamStor >>> final crumb string = yw3klV8Pa/c >>> >> https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1543010400&period2=1574546400&interval=1d&events=history&crumb=yw3klV8Pa/c >>> >>> >>> >>> JCODE: >>> NB. Navigating yahoo.com to programmatically get historical stock prices >>> NB. >>> require 'web/gethttp' >>> require 'regex' >>> >>> NB.require 'types/datetime' >>> NB. time stamp functions >>> NB. jts =: 6!:0 >>> NB. epoch =: todayno 1970 01 01 >>> NB. linuxts =: 13 : '(((toDayNo y)-epoch)*24*60*60)' >>> >>> NB. use the linux date command to create a linux time stamp >>> epochtime =: 3 : 0 >>> 2!:0 'date -jf ''%m/%d/%Y %H:%M:%S %p'' ''',y,' 05:00:00 PM'' ''+%s''' >>> ) >>> >>> NB. precision functions >>> ppq =: 9 !: 10 NB. print-precision query >>> pps =: 9 !: 11 NB. print-precision set >>> NB. I set the precision to 16 to ensure full printing of the linux >> timestamps >>> HEX=:16#.'0123456789abcdef'i.] >>> >>> xutf =: 3 : 0 >>> u: HEX tolower 2 }. y >>> ) >>> >>> crumbstr =: '"CrumbStore":{"crumb":"' >>> NB. the crumb is on the page with the link to downloading the historical >>> NB. data. If you call the correct first page you only need to search >>> NB. for the above crumbstr there will be only one. >>> getcrumb =: 3 : 0 >>> sidx =. (#crumbstr)+({: I. crumbstr E. y) >>> sstr =. (sidx + i. 30){y >>> >>> smoutput 'raw crumb string = ',(sidx + _30 + i.60){y >>> >>> eidx =. {. I. '"' E. sstr >>> >>> NB. using rxapply convert all \u00xx unicode escape sequences >>> crumb =. '(\\u[0-9a-fA-F][0-9a-fA-F][0-9a-fA-F][0-9a-fA-F])' xutf >> rxapply (i.eidx){sstr >>> ) >>> >>> financeURL =: 'https://finance.yahoo.com/quote/' NB. >> AAPL/history?p=AAPL' >>> histURL =: 'https://query1.finance.yahoo.com/v7/finance/download/' >>> NB. the histURL needs to have a ticker symbol followed by: >>> NB. ?period1=<unixts for p1>&period2=<unixts for >> p2&interval=1d&events=history&crumb=<crumbval> >>> >>> NB. >> https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1543024670&period2=1574560670&interval=1d&events=history&crumb=jZO816Y7CSK >>> >>> gethistorical=: 3 : 0 >>> 'symbol d1 d2' =. y >>> >>> NB. Create start URL at the start page with the crumb to get historical >> downloadNN >>> NB. a BASH implementation uses the commented out format >>> NB. sURL =. financeURL,symbol,'/?p=',symbol >>> >>> NB. But the link to the download of historical prices is really here: >>> sURL =. financeURL,symbol,'/history?p=',symbol >>> >>> smoutput 'start URL = ',sURL >>> >>> NB. Get the response using gethttp. --cookie-jar cookie.txt will open a >> cookie file >>> res =: '-s -c cookie.txt' gethttp sURL >>> >>> NB. tried with curl through the 2!:0 interface with the same results >>> NB. res =: 2!:0 'curl -s -c cookie.txt ',sURL >>> >>> crumb =. getcrumb res >>> smoutput 'final crumb string = ',crumb >>> >>> qstr =. '?period1=',(}:epochtime d1),'&period2=',(}:epochtime >> d2),'&interval=1d&events=history&crumb=',crumb >>> smoutput URL=. histURL,symbol,qstr >>> >>> res2 =: '-s -b cookie.txt ' gethttp URL >>> NB. res2 =: 2!:0 'curl -s -b cookie.txt ',URL >>> >>> res2 >>> ) >>> >>> ---------------------------------------------------------------------- >>> For information about J forums see http://www.jsoftware.com/forums.htm >> >> ---------------------------------------------------------------------- >> For information about J forums see http://www.jsoftware.com/forums.htm >> > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
