I researched this a bit and here is what I found: 1. oanda appears to use chunked encoding for streamed requests[1]. Curl will automatically handle this 2. There are several examples[2] available in other languages to learn from 3. The examples will trigger a new record based on the presence of the word instrument in the stream as a delimiter 4. J's web/gethttp will use curl or wget and will handle chunked encoding as a result, but I suspect it will wait for the connection to close before returning any of the message
The key decision to make in my opinion is whether to stream to a file or to parse everything in memory Streaming to a file likely makes sense in this scenario. There can be streaming use cases where it's impractical to stream to a file because the stream would consume all disk space -- then you'd have to come up with some rolling file mechanism, which would not easily be handled by curl/wget as far as I know. Parsing in memory would require writing your own HTTP get utility. You could take a look at Henry Rich's webio[3] which implements much of what would be needed Assuming you'd stream to a file, you would need to fork off a curl with the parameters to write to a file. See system/main/task.ijs and fork_jtask_ . You would then need J to continue and periodically read the file (in a loop or use a timer). The file reading routine should probably remember the file length the last time it read it and then seek to that position if the file length has changed. You could then look for the delimiter as the other examples do. That routine should also check to see if the forked curl process is still alive and if not should probably restart it (this could be because the network faulted or got disconnected for some other reason). That's probably how I would approach it. Does this help? [1] - http://developer.oanda.com/docs/v1/stream/ [2] - https://github.com/oanda/java-api-streaming/blob/master/src/main/java/JavaApiStreaming.java , https://github.com/oanda/py-api-streaming/blob/master/streaming.py [3] - http://www.jsoftware.com/jwiki/Scripts/Socket%20System?action=AttachFile&do=view&target=webio.ijs On Tue, Jul 15, 2014 at 1:00 AM, Francisco Gonzalez Velasco < [email protected]> wrote: > Hello, > > Thanks for your answer. > > I need an event for every record which comes in, and yes, I'd need to see > some code writing streaming data to file, triggering an event and reacting > to an event. > > Cheers > > Francisco > > > 2014-07-15 6:43 GMT+02:00 Raul Miller <[email protected]>: > > > I think I'd stream to file, and have J busy wait (perhaps a second of > sleep > > while waiting, to keep load down) when it's caught up. > > > > Would you want an event for every record which comes in, or would it make > > more sense to skip forward to the most recent complete event when more > data > > becomes available? Or would you want to experiment with this? > > > > Also, is that enough of an explanation? I could draft something up, I > > guess. (But not exactly right now.) > > > > Thanks, > > > > -- > > Raul > > > > > > On Tue, Jul 15, 2014 at 12:14 AM, Francisco Gonzalez Velasco < > > [email protected]> wrote: > > > > > Hello, my experience programming in J is very limited, so Inee your > > help... > > > > > > I´m using J64-602 to connect to a financial data source using their > REST > > > API, so I can get static data: > > > > > > To get the information about EUR USD in the last 5 days > > > (instrument=EUR_USD&granularity=D&count=5) I use this command: > > > > > > load 'web/gethttp' > > > > load 'convert/json' > > > > > > >>(<1 2){dec_json('-s -k -H "Authorization: Bearer > > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY"') > > > gethttp' > > > > > > > > > > https://api-fxpractice.oanda.com/v1/candles?instrument=EUR_USD&granularity=D&count=5&candleFormat=bidask > > > > ' > > > > > > > > > > > > > > ┌───────────────────────────┬───────┬───────┬───────┬───────┬───────┬───────┬────────┬────────┬──────┬──────────┐ > > > > │time │openBid│openAsk│highBid│highAsk│lowBid > > > > │lowAsk │closeBid│closeAsk│volume│complete │ > > > > > > > > > > > > > > ├───────────────────────────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┼────────┼──────┼──────────┤ > > > > > > > > > > │2014-06-10T21:00:00.000000Z│1.35431│1.35472│1.35565│1.35576│1.35212│1.35224│1.353 > > > > │1.3532 │42528 │json_true │ > > > > > > > > > > > > > > └───────────────────────────┴───────┴───────┴───────┴───────┴───────┴───────┴────────┴────────┴──────┴──────────┘ > > > > > > > > > > > > > > ┌───────────────────────────┬───────┬───────┬───────┬───────┬───────┬───────┬────────┬────────┬──────┬──────────┐ > > > > │time │openBid│openAsk│highBid│highAsk│lowBid > > > > │lowAsk │closeBid│closeAsk│volume│complete │ > > > > > > > > > > > > > > ├───────────────────────────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┼────────┼──────┼──────────┤ > > > > │2014-06-11T21:00:00.000000Z│1.353 │1.35319│1.35715│1.3573 > > > > │1.35117│1.35129│1.35494 │1.35536 │54799 │json_true │ > > > > > > > > > > > > > > └───────────────────────────┴───────┴───────┴───────┴───────┴───────┴───────┴────────┴────────┴──────┴──────────┘ > > > > > > > > > > > > > > ┌───────────────────────────┬───────┬───────┬───────┬───────┬───────┬───────┬────────┬────────┬──────┬──────────┐ > > > > │time │openBid│openAsk│highBid│highAsk│lowBid > > > > │lowAsk │closeBid│closeAsk│volume│complete │ > > > > > > > > > > > > > > ├───────────────────────────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┼────────┼──────┼──────────┤ > > > > > > > > > > │2014-06-12T21:00:00.000000Z│1.35511│1.35566│1.35786│1.35799│1.35204│1.35215│1.35374 > > > > │1.35468 │46632 │json_true │ > > > > > > > > > > > > > > └───────────────────────────┴───────┴───────┴───────┴───────┴───────┴───────┴────────┴────────┴──────┴──────────┘ > > > > > > > > > > > > > > ┌───────────────────────────┬───────┬───────┬───────┬───────┬───────┬───────┬────────┬────────┬──────┬──────────┐ > > > > │time │openBid│openAsk│highBid│highAsk│lowBid > > > > │lowAsk │closeBid│closeAsk│volume│complete │ > > > > > > > > > > > > > > ├───────────────────────────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┼────────┼──────┼──────────┤ > > > > > > > > > > │2014-06-15T21:00:00.000000Z│1.35391│1.35458│1.35791│1.35804│1.35124│1.35137│1.35733 > > > > │1.35747 │39482 │json_true │ > > > > > > > > > > > > > > └───────────────────────────┴───────┴───────┴───────┴───────┴───────┴───────┴────────┴────────┴──────┴──────────┘ > > > > > > > > > > > > > > ┌───────────────────────────┬───────┬───────┬───────┬───────┬───────┬───────┬────────┬────────┬──────┬──────────┐ > > > > │time │openBid│openAsk│highBid│highAsk│lowBid > > > > │lowAsk │closeBid│closeAsk│volume│complete │ > > > > > > > > > > > > > > ├───────────────────────────┼───────┼───────┼───────┼───────┼───────┼───────┼────────┼────────┼──────┼──────────┤ > > > > │2014-06-16T21:00:00.000000Z│1.35716│1.3573 > > > > │1.35866│1.35878│1.35526│1.35538│1.35566 │1.35577 │10173 │json_false│ > > > > > > > > > > > > > > └───────────────────────────┴───────┴───────┴───────┴───────┴───────┴───────┴────────┴────────┴──────┴──────────┘ > > > > > > > > > > > > > Now I need to know how to use streaming data... Data will be delivered > in > > > real time. > > > > > > I would use last information arrived and store data to form a database. > > > > > > The command to get the streaming data is: > > > > > > ('-s -k -H "Authorization: Bearer > > > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY"') > > > gethttp'https://stream-fxpractice.oanda.com > > > /v1/prices?accountId=abcdefg&instruments=EUR_USD' > > > > > > > > > When I use curl in the console to get streaming data the command is: > > > > > > C:\Users\Francisco>curl -k -H "Authorization: Bearer > > > XXXXXXXXXXXXXXXXXXXXXXX > > > > XXXXXXXX-YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY" " > > > > https://stream-fxpractice.oanda.com/v1/prices?accountId= > > > > < > > > > > > https://stream-fxpractice.oanda.com/v1/prices?accountId=1125870&instruments=EUR_USD > > > > > > > > abcdefg&instruments=EUR_USD > > > > < > > > > > > https://www.google.com/url?q=https%3A%2F%2Fstream-fxpractice.oanda.com%2Fv1%2Fprices%3FaccountId%3D1125870%26instruments%3DEUR_USD&sa=D&sntz=1&usg=AFQjCNEQy2PxdhBR4bYOLWuHEZSp37Xa-A > > > > > > > > " > > > > > > > > > And the incoming data is: > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.313886Z","bid":1.35744,"ask":1.35757} > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.413276Z","bid":1.35745,"ask":1.35757} > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.423912Z","bid":1.35743,"ask":1.35755} > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.518162Z","bid":1.35744,"ask":1.35755} > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.618055Z","bid":1.35743,"ask":1.35756} > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.691201Z","bid":1.35744,"ask":1.35756} > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:11.824403Z","bid":1.35745,"ask":1.35756} > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:12.250563Z","bid":1.35745,"ask":1.35758} > > > > {"heartbeat":{"time":"2014-06-17T08:50:12.547586Z"}} > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:12.611163Z","bid":1.35746,"ask":1.35758} > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:13.168306Z","bid":1.35746,"ask":1.35757} > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:13.527477Z","bid":1.35745,"ask":1.35757} > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:14.009987Z","bid":1.35746,"ask":1.35756} > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:14.109158Z","bid":1.35745,"ask":1.35757} > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:14.228386Z","bid":1.35743,"ask":1.35755} > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:14.314567Z","bid":1.35744,"ask":1.35755} > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:14.912960Z","bid":1.35742,"ask":1.35753} > > > > {"heartbeat":{"time":"2014-06-17T08:50:15.036929Z"}} > > > > > > > > > > > > > > {"instrument":"EUR_USD","time":"2014-06-17T08:50:15.214964Z","bid":1.3574,"ask":1.35751} > > > > > > > > > Token and account number have been changed > > > > > > > > > 1) I'd like to know how to have a variable holding the most recent data > > > sent by the data provider. And an event triggering when a new value > > arrives > > > so actions can be taken depending on the values. > > > 2) I'd like to write the incoming data to disk (as a csv file, JDB,...) > > > for persistance. > > > > > > Cheers > > > > > > Francisco > > > ---------------------------------------------------------------------- > > > For information about J forums see http://www.jsoftware.com/forums.htm > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
