Hi, yes, there's definitely a market for Apache Spark and financial institutions, I can't provide specific details but to answer your survey: "yes" and "more than a few GB!"
Here are a couple of examples showing Spark with financial data, full disclosure that I work for IBM, I'm sure there are lots more examples you can find too: https://www.youtube.com/watch?v=VWBNoIwGEjo shows how Spark can be used with simple sentiment analysis to figure out correlations between real world events and stock market changes. The Spark specific part is from 3 04 until the 8th minute https://www.youtube.com/watch?v=sDmWcuO5Rk8 is a similar example where Spark is again used with sentiment analysis. One could also analyse financial data to identify trends, I think a lot of the machine learning APIs will be useful here e.g. logistic regression with many features could be used to decide whether or not an investment is a good idea based on training data (so we'd look at real outcomes from previous speculations) In both cases you can see Spark is a very important component for performing the calculations with financial data. I also know that Goldman Sachs mentioned they are interested in Spark, one talk is at https://www.youtube.com/watch?v=HWwAoTK2YrQ, so this is more evidence of the financial industries paying attention to big data and Spark. Regarding your app: I expected it to be similar to the first example where the signals you mention are real world events (e.g. the fed lowers interest rates or companies are rumoured to either be about to float or be acquired). At the 4 30 part I think you actually identify previous index values and extrapolate what they are likely to become using, so in theory your system would become more accurate over time although would going off indexes alone be sufficient (if indeed this is what you're doing). I think you'd want to combine this with real world speculation/news to figure out *why* the price is likely to change, how much by and in which direction. I agree that Apache Spark can be just the right tool for doing the heavy lifting required for analysis, computation and modelling of big data so looking forward to future Spark work in this area, and I wonder how we as Spark developers can make it easier/more powerful for Spark users to do so From: "Taotao.Li" <charles.up...@gmail.com> To: user <user@spark.apache.org> Date: 30/08/2016 14:14 Subject: Hi, guys, does anyone use Spark in finance market? Hi, guys, I'm a quant engineer in China, and I believe it's very promising when using Spark in the financial market. But I didn't find cases which combine spark and finance. So here I wanna do a small survey: do you guys use Spark in financial market related project? if yes, how large data was fed in your spark application? thanks a lot. ___________________ A little ad, I attended IBM Spark Hackathon, which is here : http://apachespark.devpost.com/ , and I submitted a small application, which will be used in my strategies, hope you guys and give me a vote and some suggestions on how to use spark in financial market, to discover some trade opportunity. here is my small app: http://devpost.com/software/spark-in-finance-quantitative-investing thanks a lot. -- ___________________ Quant | Engineer | Boy ___________________ blog: http://litaotao.github.io github: www.github.com/litaotao Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
