The following J script, patterned after Brad Lucas’ Bash script (http://blog.bradlucas.com/posts/2017-06-02-new-yahoo-finance-quote-download-url/ <http://blog.bradlucas.com/posts/2017-06-02-new-yahoo-finance-quote-download-url/>), doesn’t seem to work. I won’t go into the history of the yahoo and the change in their website except to say that by making a double access with curl and pulling out elements of the first response to use in the second access you should be able to programmatically get historical price quotes.
I believe I have set up the URLs properly. I am using cookies with web/gethttp function. I see the cookie file get created and I receive a first response from the yahoo website. From there you need to find the ‘crumb’ in the response and use that to build a URL to request the historical stock prices. The crumb can sometimes contain \u00xx unicode escape characters. These need to be translated into a character. Bash has a built in echo command (different than the linux echo command) that will translate this properly. In J I use reapply to translate these sequences. The Bash version redirects the output of the second curl call to a file. I am just trying to get the data to print in Jqt where I run the script. I will show the output and the J script will follow. I tried this with 2!:0 foreign conjunction to run curl directly and have the same problem. The first call to curl gets an appropriate response and the second call gets an error stating invalid cookie. This error is not displayed in J I modified the code to log the errors to a file and found it there. OUTPUT: gethistorical 'AAPL';'11/23/2018';'11/23/2019' start URL = https://finance.yahoo.com/quote/AAPL/history?p=AAPL raw crumb string = "}}}}},"CrumbStore":{"crumb":"34jVbQ1iOCI"},"StreamStore":{" final crumb string = 34jVbQ1iOCI https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1543010400&period2=1574546400&interval=1d&events=history&crumb=34jVbQ1iOCI OUTPUT (with unicode escape sequence): gethistorical 'AAPL';'11/23/2018';'11/23/2019' start URL = https://finance.yahoo.com/quote/AAPL/?p=AAPL raw crumb string = "}}}}},"CrumbStore":{"crumb":"yw3klV8Pa\u002Fc"},"StreamStor final crumb string = yw3klV8Pa/c https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1543010400&period2=1574546400&interval=1d&events=history&crumb=yw3klV8Pa/c JCODE: NB. Navigating yahoo.com to programmatically get historical stock prices NB. require 'web/gethttp' require 'regex' NB.require 'types/datetime' NB. time stamp functions NB. jts =: 6!:0 NB. epoch =: todayno 1970 01 01 NB. linuxts =: 13 : '(((toDayNo y)-epoch)*24*60*60)' NB. use the linux date command to create a linux time stamp epochtime =: 3 : 0 2!:0 'date -jf ''%m/%d/%Y %H:%M:%S %p'' ''',y,' 05:00:00 PM'' ''+%s''' ) NB. precision functions ppq =: 9 !: 10 NB. print-precision query pps =: 9 !: 11 NB. print-precision set NB. I set the precision to 16 to ensure full printing of the linux timestamps HEX=:16#.'0123456789abcdef'i.] xutf =: 3 : 0 u: HEX tolower 2 }. y ) crumbstr =: '"CrumbStore":{"crumb":"' NB. the crumb is on the page with the link to downloading the historical NB. data. If you call the correct first page you only need to search NB. for the above crumbstr there will be only one. getcrumb =: 3 : 0 sidx =. (#crumbstr)+({: I. crumbstr E. y) sstr =. (sidx + i. 30){y smoutput 'raw crumb string = ',(sidx + _30 + i.60){y eidx =. {. I. '"' E. sstr NB. using rxapply convert all \u00xx unicode escape sequences crumb =. '(\\u[0-9a-fA-F][0-9a-fA-F][0-9a-fA-F][0-9a-fA-F])' xutf rxapply (i.eidx){sstr ) financeURL =: 'https://finance.yahoo.com/quote/' NB. AAPL/history?p=AAPL' histURL =: 'https://query1.finance.yahoo.com/v7/finance/download/' NB. the histURL needs to have a ticker symbol followed by: NB. ?period1=<unixts for p1>&period2=<unixts for p2&interval=1d&events=history&crumb=<crumbval> NB. https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1543024670&period2=1574560670&interval=1d&events=history&crumb=jZO816Y7CSK gethistorical=: 3 : 0 'symbol d1 d2' =. y NB. Create start URL at the start page with the crumb to get historical downloadNN NB. a BASH implementation uses the commented out format NB. sURL =. financeURL,symbol,'/?p=',symbol NB. But the link to the download of historical prices is really here: sURL =. financeURL,symbol,'/history?p=',symbol smoutput 'start URL = ',sURL NB. Get the response using gethttp. --cookie-jar cookie.txt will open a cookie file res =: '-s -c cookie.txt' gethttp sURL NB. tried with curl through the 2!:0 interface with the same results NB. res =: 2!:0 'curl -s -c cookie.txt ',sURL crumb =. getcrumb res smoutput 'final crumb string = ',crumb qstr =. '?period1=',(}:epochtime d1),'&period2=',(}:epochtime d2),'&interval=1d&events=history&crumb=',crumb smoutput URL=. histURL,symbol,qstr res2 =: '-s -b cookie.txt ' gethttp URL NB. res2 =: 2!:0 'curl -s -b cookie.txt ',URL res2 ) ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
