I personally find it disappointing that a big chuck of Spark's design and development is happening behind closed curtains. It makes it harder than necessary for me to work with Spark. We had to improvise in the recent weeks a temporary solution for reading from Kafka (from Structured Streaming) to unblock our development, and I feed that if the design and development of that feature was done in the open, it would have saved us a lot of hassle (and would reduce the refactoring of our code base).
It hard not compare it to other Apache projects - for example, I believe most of the Apache Kafka full-time contributors work at a single company, but they manage as a community to have a very transparent design and development process, which seems to work great. Ofir Manor Co-Founder & CTO | Equalum Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io On Mon, Aug 29, 2016 at 10:39 PM, Fred Reiss <freiss....@gmail.com> wrote: > I think that the community really needs some feedback on the progress of > this very important task. Many existing Spark Streaming applications can't > be ported to Structured Streaming without Kafka support. > > Is there a design document somewhere? Or can someone from the DataBricks > team break down the existing monolithic JIRA issue into smaller steps that > reflect the current development plan? > > Fred > > > On Sat, Aug 27, 2016 at 2:32 PM, Koert Kuipers <ko...@tresata.com> wrote: > >> thats great >> >> is this effort happening anywhere that is publicly visible? github? >> >> On Tue, Aug 16, 2016 at 2:04 AM, Reynold Xin <r...@databricks.com> wrote: >> >>> We (the team at Databricks) are working on one currently. >>> >>> >>> On Mon, Aug 15, 2016 at 7:26 PM, Cody Koeninger <c...@koeninger.org> >>> wrote: >>> >>>> https://issues.apache.org/jira/browse/SPARK-15406 >>>> >>>> I'm not working on it (yet?), never got an answer to the question of >>>> who was planning to work on it. >>>> >>>> On Mon, Aug 15, 2016 at 9:12 PM, Guo, Chenzhao <chenzhao....@intel.com> >>>> wrote: >>>> > Hi all, >>>> > >>>> > >>>> > >>>> > I’m trying to write Structured Streaming test code and will deal with >>>> Kafka >>>> > source. Currently Spark 2.0 doesn’t support Kafka sources/sinks. >>>> > >>>> > >>>> > >>>> > I found some Databricks slides saying that Kafka sources/sinks will be >>>> > implemented in Spark 2.0, so is there anybody working on this? And >>>> when will >>>> > it be released? >>>> > >>>> > >>>> > >>>> > Thanks, >>>> > >>>> > Chenzhao Guo >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>> >>>> >>> >> >
