Hi Ofir, Thanks for the elaborated answer. I have read both documents, where they do a light touch on infinite Dataframes/Datasets. However, they do not go in depth as regards to how existing transformations on DStreams, for example, will be transformed into the Dataset APIs. I've been browsing the 2.0 branch and have yet been able to understand how they correlate.
Also, placing SparkSession in the sql package seems like a peculiar choice, since this is going to be the global abstraction over SparkContext/StreamingContext from now on. On Sun, May 15, 2016, 23:42 Ofir Manor <ofir.ma...@equalum.io> wrote: > Hi Yuval, > let me share my understanding based on similar questions I had. > First, Spark 2.x aims to replace a whole bunch of its APIs with just two > main ones - SparkSession (replacing Hive/SQL/Spark Context) and Dataset > (merging of Dataset and Dataframe - which is why it inherits all the > SparkSQL goodness), while RDD seems as a low-level API only for special > cases. The new Dataset should also support both batch and streaming - > replacing (eventually) DStream as well. See the design docs in SPARK-13485 > (unified API) and SPARK-8360 (StructuredStreaming) for a good intro. > However, as you noted, not all will be fully delivered in 2.0. For > example, it seems that streaming from / to Kafka using StructuredStreaming > didn't make it (so far?) to 2.0 (which is a showstopper for me). > Anyway, as far as I understand, you should be able to apply stateful > operators (non-RDD) on Datasets (for example, the new event-time window > processing SPARK-8360). The gap I see is mostly limited streaming sources / > sinks migrated to the new (richer) API and semantics. > Anyway, I'm pretty sure once 2.0 gets to RC, the documentation and > examples will align with the current offering... > > > Ofir Manor > > Co-Founder & CTO | Equalum > > Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io > > On Sun, May 15, 2016 at 1:52 PM, Yuval.Itzchakov <yuva...@gmail.com> > wrote: > >> I've been reading/watching videos about the upcoming Spark 2.0 release >> which >> brings us Structured Streaming. One thing I've yet to understand is how >> this >> relates to the current state of working with Streaming in Spark with the >> DStream abstraction. >> >> All examples I can find, in the Spark repository/different videos is >> someone >> streaming local JSON files or reading from HDFS/S3/SQL. Also, when >> browsing >> the source, SparkSession seems to be defined inside org.apache.spark.sql, >> so >> this gives me a hunch that this is somehow all related to SQL and the >> likes, >> and not really to DStreams. >> >> What I'm failing to understand is: Will this feature impact how we do >> Streaming today? Will I be able to consume a Kafka source in a streaming >> fashion (like we do today when we open a stream using KafkaUtils)? Will we >> be able to do state-full operations on a Dataset[T] like we do today using >> MapWithStateRDD? Or will there be a subset of operations that the catalyst >> optimizer can understand such as aggregate and such? >> >> I'd be happy anyone could shed some light on this. >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Structured-Streaming-in-Spark-2-0-and-DStreams-tp26959.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >>
