Jags, I should have been more specific. I am referring to what I read at http://snappydatainc.github.io/snappydata/streamingWithSQL/, especially the Streaming Tables part. It roughly coincides with the Streaming DataFrames outlined here https://docs.google.com/document/d/1NHKdRSNCbCmJbinLmZuqNA1Pt6CGpFnLVRbzuDUcZVM/edit#heading=h.ff0opfdo6q1h. I don’t if I’m wrong, but they both sound very similar. That’s why I posed this question.
Thanks, Ben > On Jul 6, 2016, at 7:03 AM, Jags Ramnarayan <jramnara...@snappydata.io> wrote: > > Ben, > Note that Snappydata's primary objective is to be a distributed in-memory > DB for mixed workloads (i.e. streaming with transactions and analytic > queries). On the other hand, Spark, till date, is primarily designed as a > processing engine over myriad storage engines (SnappyData being one). So, the > marriage is quite complementary. The difference compared to other stores is > that SnappyData realizes its solution by deeply integrating and collocating > with Spark (i.e. share spark executor memory/resources with the store) > avoiding serializations and shuffle in many situations. > > On your specific thought about being similar to Structured streaming, a > better discussion could be a comparison to the recently introduced State > store > <https://docs.google.com/document/d/1-ncawFx8JS5Zyfq1HAEGBx56RDet9wfVp_hDM8ZL254/edit#heading=h.2h7zw4ru3nw7> > (perhaps this is what you meant). > It proposes a KV store for streaming aggregations with support for updates. > The proposed API will, at some point, be pluggable so vendors can easily > support alternate implementations to storage, not just HDFS(default store in > proposed State store). > > > ----- > Jags > SnappyData blog <http://www.snappydata.io/blog> > Download binary, source <https://github.com/SnappyDataInc/snappydata> > > > On Wed, Jul 6, 2016 at 12:49 AM, Benjamin Kim <bbuil...@gmail.com > <mailto:bbuil...@gmail.com>> wrote: > I recently got a sales email from SnappyData, and after reading the > documentation about what they offer, it sounds very similar to what > Structured Streaming will offer w/o the underlying in-memory, spill-to-disk, > CRUD compliant data storage in SnappyData. I was wondering if Structured > Streaming is trying to achieve the same on its own or is SnappyData > contributing Streaming extensions that they built to the Spark project. > Lastly, what does the Spark community think of this so-called “Spark Data > Store”? > > Thanks, > Ben > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > <mailto:user-unsubscr...@spark.apache.org> > >
