Haven't got time to look at your code in detail, but stabbing blindly in the dark - have you tried double quoting your URL? See the following StackOverflow question and answer: https://stackoverflow.com/questions/55753504/reading-csv-file-from-a-a-secured-link-with-j
On Mon, Nov 25, 2019 at 3:36 PM Henry Rich <henryhr...@gmail.com> wrote: > Do you have an example of a script where you can see correct data going > in and a bad result coming out? > > Henry Rich > > On 11/24/2019 9:30 PM, Thomas McGuire wrote: > > The following J script, patterned after Brad Lucas’ Bash script ( > http://blog.bradlucas.com/posts/2017-06-02-new-yahoo-finance-quote-download-url/ > < > http://blog.bradlucas.com/posts/2017-06-02-new-yahoo-finance-quote-download-url/>), > doesn’t seem to work. I won’t go into the history of the yahoo and the > change in their website except to say that by making a double access with > curl and pulling out elements of the first response to use in the second > access you should be able to programmatically get historical price quotes. > > > > I believe I have set up the URLs properly. I am using cookies with > web/gethttp function. I see the cookie file get created and I receive a > first response from the yahoo website. From there you need to find the > ‘crumb’ in the response and use that to build a URL to request the > historical stock prices. The crumb can sometimes contain \u00xx unicode > escape characters. These need to be translated into a character. Bash has a > built in echo command (different than the linux echo command) that will > translate this properly. In J I use reapply to translate these sequences. > The Bash version redirects the output of the second curl call to a file. I > am just trying to get the data to print in Jqt where I run the script. > > > > I will show the output and the J script will follow. I tried this with > 2!:0 foreign conjunction to run curl directly and have the same problem. > The first call to curl gets an appropriate response and the second call > gets an error stating invalid cookie. This error is not displayed in J I > modified the code to log the errors to a file and found it there. > > > > OUTPUT: > > gethistorical 'AAPL';'11/23/2018';'11/23/2019' > > start URL = https://finance.yahoo.com/quote/AAPL/history?p=AAPL > > raw crumb string = > "}}}}},"CrumbStore":{"crumb":"34jVbQ1iOCI"},"StreamStore":{" > > final crumb string = 34jVbQ1iOCI > > > https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1543010400&period2=1574546400&interval=1d&events=history&crumb=34jVbQ1iOCI > > > > OUTPUT (with unicode escape sequence): > > gethistorical 'AAPL';'11/23/2018';'11/23/2019' > > start URL = https://finance.yahoo.com/quote/AAPL/?p=AAPL > > raw crumb string = > "}}}}},"CrumbStore":{"crumb":"yw3klV8Pa\u002Fc"},"StreamStor > > final crumb string = yw3klV8Pa/c > > > https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1543010400&period2=1574546400&interval=1d&events=history&crumb=yw3klV8Pa/c > > > > > > > > JCODE: > > NB. Navigating yahoo.com to programmatically get historical stock prices > > NB. > > require 'web/gethttp' > > require 'regex' > > > > NB.require 'types/datetime' > > NB. time stamp functions > > NB. jts =: 6!:0 > > NB. epoch =: todayno 1970 01 01 > > NB. linuxts =: 13 : '(((toDayNo y)-epoch)*24*60*60)' > > > > NB. use the linux date command to create a linux time stamp > > epochtime =: 3 : 0 > > 2!:0 'date -jf ''%m/%d/%Y %H:%M:%S %p'' ''',y,' 05:00:00 PM'' ''+%s''' > > ) > > > > NB. precision functions > > ppq =: 9 !: 10 NB. print-precision query > > pps =: 9 !: 11 NB. print-precision set > > NB. I set the precision to 16 to ensure full printing of the linux > timestamps > > HEX=:16#.'0123456789abcdef'i.] > > > > xutf =: 3 : 0 > > u: HEX tolower 2 }. y > > ) > > > > crumbstr =: '"CrumbStore":{"crumb":"' > > NB. the crumb is on the page with the link to downloading the historical > > NB. data. If you call the correct first page you only need to search > > NB. for the above crumbstr there will be only one. > > getcrumb =: 3 : 0 > > sidx =. (#crumbstr)+({: I. crumbstr E. y) > > sstr =. (sidx + i. 30){y > > > > smoutput 'raw crumb string = ',(sidx + _30 + i.60){y > > > > eidx =. {. I. '"' E. sstr > > > > NB. using rxapply convert all \u00xx unicode escape sequences > > crumb =. '(\\u[0-9a-fA-F][0-9a-fA-F][0-9a-fA-F][0-9a-fA-F])' xutf > rxapply (i.eidx){sstr > > ) > > > > financeURL =: 'https://finance.yahoo.com/quote/' NB. > AAPL/history?p=AAPL' > > histURL =: 'https://query1.finance.yahoo.com/v7/finance/download/' > > NB. the histURL needs to have a ticker symbol followed by: > > NB. ?period1=<unixts for p1>&period2=<unixts for > p2&interval=1d&events=history&crumb=<crumbval> > > > > NB. > https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1543024670&period2=1574560670&interval=1d&events=history&crumb=jZO816Y7CSK > > > > gethistorical=: 3 : 0 > > 'symbol d1 d2' =. y > > > > NB. Create start URL at the start page with the crumb to get historical > downloadNN > > NB. a BASH implementation uses the commented out format > > NB. sURL =. financeURL,symbol,'/?p=',symbol > > > > NB. But the link to the download of historical prices is really here: > > sURL =. financeURL,symbol,'/history?p=',symbol > > > > smoutput 'start URL = ',sURL > > > > NB. Get the response using gethttp. --cookie-jar cookie.txt will open a > cookie file > > res =: '-s -c cookie.txt' gethttp sURL > > > > NB. tried with curl through the 2!:0 interface with the same results > > NB. res =: 2!:0 'curl -s -c cookie.txt ',sURL > > > > crumb =. getcrumb res > > smoutput 'final crumb string = ',crumb > > > > qstr =. '?period1=',(}:epochtime d1),'&period2=',(}:epochtime > d2),'&interval=1d&events=history&crumb=',crumb > > smoutput URL=. histURL,symbol,qstr > > > > res2 =: '-s -b cookie.txt ' gethttp URL > > NB. res2 =: 2!:0 'curl -s -b cookie.txt ',URL > > > > res2 > > ) > > > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
