I have a curiosity question. These forever/unlimited DataFrames/DataSets will persist and be query capable. I still am foggy about how this data will be stored. As far as I know, memory is finite. Will the data be spilled to disk and be retrievable if the query spans data not in memory? Is Tachyon (Alluxio), HDFS (Parquet), NoSQL (HBase, Cassandra), RDBMS (PostgreSQL, MySQL), Object Store (S3, Swift), or any else I can’t think of going to be the underlying near real-time storage system?
Thanks, Ben > On May 15, 2016, at 3:36 PM, Yuval Itzchakov <yuva...@gmail.com> wrote: > > Hi Ofir, > Thanks for the elaborated answer. I have read both documents, where they do a > light touch on infinite Dataframes/Datasets. However, they do not go in depth > as regards to how existing transformations on DStreams, for example, will be > transformed into the Dataset APIs. I've been browsing the 2.0 branch and have > yet been able to understand how they correlate. > > Also, placing SparkSession in the sql package seems like a peculiar choice, > since this is going to be the global abstraction over > SparkContext/StreamingContext from now on. > > On Sun, May 15, 2016, 23:42 Ofir Manor <ofir.ma...@equalum.io > <mailto:ofir.ma...@equalum.io>> wrote: > Hi Yuval, > let me share my understanding based on similar questions I had. > First, Spark 2.x aims to replace a whole bunch of its APIs with just two main > ones - SparkSession (replacing Hive/SQL/Spark Context) and Dataset (merging > of Dataset and Dataframe - which is why it inherits all the SparkSQL > goodness), while RDD seems as a low-level API only for special cases. The new > Dataset should also support both batch and streaming - replacing (eventually) > DStream as well. See the design docs in SPARK-13485 (unified API) and > SPARK-8360 (StructuredStreaming) for a good intro. > However, as you noted, not all will be fully delivered in 2.0. For example, > it seems that streaming from / to Kafka using StructuredStreaming didn't make > it (so far?) to 2.0 (which is a showstopper for me). > Anyway, as far as I understand, you should be able to apply stateful > operators (non-RDD) on Datasets (for example, the new event-time window > processing SPARK-8360). The gap I see is mostly limited streaming sources / > sinks migrated to the new (richer) API and semantics. > Anyway, I'm pretty sure once 2.0 gets to RC, the documentation and examples > will align with the current offering... > > > Ofir Manor > > Co-Founder & CTO | Equalum > > > Mobile: +972-54-7801286 <tel:%2B972-54-7801286> | Email: > ofir.ma...@equalum.io <mailto:ofir.ma...@equalum.io> > On Sun, May 15, 2016 at 1:52 PM, Yuval.Itzchakov <yuva...@gmail.com > <mailto:yuva...@gmail.com>> wrote: > I've been reading/watching videos about the upcoming Spark 2.0 release which > brings us Structured Streaming. One thing I've yet to understand is how this > relates to the current state of working with Streaming in Spark with the > DStream abstraction. > > All examples I can find, in the Spark repository/different videos is someone > streaming local JSON files or reading from HDFS/S3/SQL. Also, when browsing > the source, SparkSession seems to be defined inside org.apache.spark.sql, so > this gives me a hunch that this is somehow all related to SQL and the likes, > and not really to DStreams. > > What I'm failing to understand is: Will this feature impact how we do > Streaming today? Will I be able to consume a Kafka source in a streaming > fashion (like we do today when we open a stream using KafkaUtils)? Will we > be able to do state-full operations on a Dataset[T] like we do today using > MapWithStateRDD? Or will there be a subset of operations that the catalyst > optimizer can understand such as aggregate and such? > > I'd be happy anyone could shed some light on this. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Structured-Streaming-in-Spark-2-0-and-DStreams-tp26959.html > > <http://apache-spark-user-list.1001560.n3.nabble.com/Structured-Streaming-in-Spark-2-0-and-DStreams-tp26959.html> > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > <mailto:user-unsubscr...@spark.apache.org> > For additional commands, e-mail: user-h...@spark.apache.org > <mailto:user-h...@spark.apache.org> >
