AFAIK, the underlying data represented under the DataSet[T] abstraction will be formatted in Tachyon under the hood, but as with RDD's if needed they will be spilled to local disk on the worker of needed.
On Mon, May 16, 2016, 19:47 Benjamin Kim <bbuil...@gmail.com> wrote: > I have a curiosity question. These forever/unlimited DataFrames/DataSets > will persist and be query capable. I still am foggy about how this data > will be stored. As far as I know, memory is finite. Will the data be > spilled to disk and be retrievable if the query spans data not in memory? > Is Tachyon (Alluxio), HDFS (Parquet), NoSQL (HBase, Cassandra), RDBMS > (PostgreSQL, MySQL), Object Store (S3, Swift), or any else I can’t think of > going to be the underlying near real-time storage system? > > Thanks, > Ben > > > On May 15, 2016, at 3:36 PM, Yuval Itzchakov <yuva...@gmail.com> wrote: > > Hi Ofir, > Thanks for the elaborated answer. I have read both documents, where they > do a light touch on infinite Dataframes/Datasets. However, they do not go > in depth as regards to how existing transformations on DStreams, for > example, will be transformed into the Dataset APIs. I've been browsing the > 2.0 branch and have yet been able to understand how they correlate. > > Also, placing SparkSession in the sql package seems like a peculiar > choice, since this is going to be the global abstraction over > SparkContext/StreamingContext from now on. > > On Sun, May 15, 2016, 23:42 Ofir Manor <ofir.ma...@equalum.io> wrote: > >> Hi Yuval, >> let me share my understanding based on similar questions I had. >> First, Spark 2.x aims to replace a whole bunch of its APIs with just two >> main ones - SparkSession (replacing Hive/SQL/Spark Context) and Dataset >> (merging of Dataset and Dataframe - which is why it inherits all the >> SparkSQL goodness), while RDD seems as a low-level API only for special >> cases. The new Dataset should also support both batch and streaming - >> replacing (eventually) DStream as well. See the design docs in SPARK-13485 >> (unified API) and SPARK-8360 (StructuredStreaming) for a good intro. >> However, as you noted, not all will be fully delivered in 2.0. For >> example, it seems that streaming from / to Kafka using StructuredStreaming >> didn't make it (so far?) to 2.0 (which is a showstopper for me). >> Anyway, as far as I understand, you should be able to apply stateful >> operators (non-RDD) on Datasets (for example, the new event-time window >> processing SPARK-8360). The gap I see is mostly limited streaming sources / >> sinks migrated to the new (richer) API and semantics. >> Anyway, I'm pretty sure once 2.0 gets to RC, the documentation and >> examples will align with the current offering... >> >> >> Ofir Manor >> >> Co-Founder & CTO | Equalum >> >> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io >> >> On Sun, May 15, 2016 at 1:52 PM, Yuval.Itzchakov <yuva...@gmail.com> >> wrote: >> >>> I've been reading/watching videos about the upcoming Spark 2.0 release >>> which >>> brings us Structured Streaming. One thing I've yet to understand is how >>> this >>> relates to the current state of working with Streaming in Spark with the >>> DStream abstraction. >>> >>> All examples I can find, in the Spark repository/different videos is >>> someone >>> streaming local JSON files or reading from HDFS/S3/SQL. Also, when >>> browsing >>> the source, SparkSession seems to be defined inside >>> org.apache.spark.sql, so >>> this gives me a hunch that this is somehow all related to SQL and the >>> likes, >>> and not really to DStreams. >>> >>> What I'm failing to understand is: Will this feature impact how we do >>> Streaming today? Will I be able to consume a Kafka source in a streaming >>> fashion (like we do today when we open a stream using KafkaUtils)? Will >>> we >>> be able to do state-full operations on a Dataset[T] like we do today >>> using >>> MapWithStateRDD? Or will there be a subset of operations that the >>> catalyst >>> optimizer can understand such as aggregate and such? >>> >>> I'd be happy anyone could shed some light on this. >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Structured-Streaming-in-Spark-2-0-and-DStreams-tp26959.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com >>> <http://nabble.com>. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >
