Keeping in mind Spark is a parallel computing engine, Spark does not change your data infrastructure/data architecture. These days it's relatively convenient to read data from a variety of sources (S3, HDFS, Cassandra, ...) and ditto on the output side.
For example, for one of my use-cases, I store 10's of gigs of time-series data in Cassandra. It just so happens I like to analyze all of it at once using Spark, which writes a very nice, small text file table of results I look at using Python/Pandas, in a Jupyter notebook, on a laptop. If we didn't have Spark, I'd still be doing the input side (Cassandra) and output side (small text file, ingestible by a laptop) the same way. The only difference would be, instead of importing and processing in Spark, my fictional group of 5,000 assistants would each download a portion of the data into their Excel spreadsheet, then have a big meeting to produce my small text file. So my view is the nature of your data and specific objectives determine your infrastructure and architecture, not the presence or absence of Spark. On Sat, Dec 3, 2016 at 10:59 AM, Vasu Gourabathina <vgour...@gmail.com> wrote: > Hi, > > I know this is a broad question. If this is not the right forum, > appreciate if you can point to other sites/areas that may be helpful. > > Before posing this question, I did use our friend Google, but sanitizing > the query results from my need angle hasn't been easy. > > Who I am: > - Have done data processing and analytics, but relatively new to Spark > world > > What I am looking for: > - Architecture/Design of a ML system using Spark > - In particular, looking for best practices that can support/bridge both > Engineering and Data Science teams > > Engineering: > - Build a system that has typical engineering needs, data processing, > scalability, reliability, availability, fault-tolerance etc. > - System monitoring etc. > Data Science: > - Build a system for Data Science team to do data exploration activities > - Develop models using supervised learning and tweak models > > Data: > - Batch and incremental updates - mostly structured or semi-structured > (some data from transaction systems, weblogs, click stream etc.) > - Steaming, in near term, but not to begin with > > Data Storage: > - Data is expected to grow on a daily basis...so, system should be able > to support and handle big data > - May be, after further analysis, there might be a possibility/need to > archive some of the data...it all depends on how the ML models were built > and results were stored/used for future usage > > Data Analysis: > - Obvious data related aspects, such as data cleansing, data > transformation, data partitioning etc > - May be run models on windows of data. For example: last 1-year, > 2-years etc. > > ML models: > - Ability to store model versions and previous results > - Compare results of different variants of models > > Consumers: > - RESTful webservice clients to look at the results > > *So, the questions I have are:* > 1) Are there architectural and design patterns that I can use based on > industry best-practices. In particular: > - data ingestion > - data storage (for eg. go with HDFS or not) > - data partitioning, especially in Spark world > - running parallel ML models and combining results etc. > - consumption of final results by clients (for eg. by pushing > results to Cassandra, NoSQL dbs etc.) > > Again, I know this is a broad question....Pointers to some best-practices > in some of the areas, if not all, would be highly appreciated. Open to > purchase any books that may have relevant information. > > Thanks much folks, > Vasu. > >
