> I personally find it disappointing that a big chuck of Spark's design and development is happening behind closed curtains.
I'm not too familiar with Streaming, but I see design docs and proposals for ML and SQL published here and on JIRA all the time, and they are discussed extensively. For example, here are some ML JIRAs with extensive design discussions: SPARK-6725 <https://issues.apache.org/jira/browse/SPARK-6725>, SPARK-13944 <https://issues.apache.org/jira/browse/SPARK-13944>, SPARK-16365 <https://issues.apache.org/jira/browse/SPARK-16365> Nick On Tue, Aug 30, 2016 at 11:10 AM Cody Koeninger <c...@koeninger.org> wrote: > Not that I wouldn't rather have more open communication around this > issue...but what are people actually expecting to get out of > structured streaming with regard to Kafka? > > There aren't any realistic pushdown-type optimizations available, and > from what I could tell the last time I looked at structured streaming, > resolving the event time vs processing time issue was still a ways > off. > > On Tue, Aug 30, 2016 at 1:56 AM, Ofir Manor <ofir.ma...@equalum.io> wrote: > > I personally find it disappointing that a big chuck of Spark's design and > > development is happening behind closed curtains. It makes it harder than > > necessary for me to work with Spark. We had to improvise in the recent > weeks > > a temporary solution for reading from Kafka (from Structured Streaming) > to > > unblock our development, and I feed that if the design and development of > > that feature was done in the open, it would have saved us a lot of hassle > > (and would reduce the refactoring of our code base). > > > > It hard not compare it to other Apache projects - for example, I believe > > most of the Apache Kafka full-time contributors work at a single company, > > but they manage as a community to have a very transparent design and > > development process, which seems to work great. > > > > Ofir Manor > > > > Co-Founder & CTO | Equalum > > > > Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io > > > > > > On Mon, Aug 29, 2016 at 10:39 PM, Fred Reiss <freiss....@gmail.com> > wrote: > >> > >> I think that the community really needs some feedback on the progress of > >> this very important task. Many existing Spark Streaming applications > can't > >> be ported to Structured Streaming without Kafka support. > >> > >> Is there a design document somewhere? Or can someone from the > DataBricks > >> team break down the existing monolithic JIRA issue into smaller steps > that > >> reflect the current development plan? > >> > >> Fred > >> > >> > >> On Sat, Aug 27, 2016 at 2:32 PM, Koert Kuipers <ko...@tresata.com> > wrote: > >>> > >>> thats great > >>> > >>> is this effort happening anywhere that is publicly visible? github? > >>> > >>> On Tue, Aug 16, 2016 at 2:04 AM, Reynold Xin <r...@databricks.com> > wrote: > >>>> > >>>> We (the team at Databricks) are working on one currently. > >>>> > >>>> > >>>> On Mon, Aug 15, 2016 at 7:26 PM, Cody Koeninger <c...@koeninger.org> > >>>> wrote: > >>>>> > >>>>> https://issues.apache.org/jira/browse/SPARK-15406 > >>>>> > >>>>> I'm not working on it (yet?), never got an answer to the question of > >>>>> who was planning to work on it. > >>>>> > >>>>> On Mon, Aug 15, 2016 at 9:12 PM, Guo, Chenzhao < > chenzhao....@intel.com> > >>>>> wrote: > >>>>> > Hi all, > >>>>> > > >>>>> > > >>>>> > > >>>>> > I’m trying to write Structured Streaming test code and will deal > with > >>>>> > Kafka > >>>>> > source. Currently Spark 2.0 doesn’t support Kafka sources/sinks. > >>>>> > > >>>>> > > >>>>> > > >>>>> > I found some Databricks slides saying that Kafka sources/sinks will > >>>>> > be > >>>>> > implemented in Spark 2.0, so is there anybody working on this? And > >>>>> > when will > >>>>> > it be released? > >>>>> > > >>>>> > > >>>>> > > >>>>> > Thanks, > >>>>> > > >>>>> > Chenzhao Guo > >>>>> > >>>>> --------------------------------------------------------------------- > >>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >>>>> > >>>> > >>> > >> > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
