Good point Ismael :D
On Mon, Oct 24, 2016 at 11:07 PM, Ismael Juma <ism...@juma.me.uk> wrote:
> If we want to start a vote on this, I suggest starting a [VOTE] thread
> instead of mentioning it in the middle of a very long [DISCUSS] thread. :)
>
> Ismael
>
> On Mon, Oct 24, 2016 at 9:46 PM, Ben Davison <ben.davi...@7digital.com>
> wrote:
>
> > This KIP has got muddy on differing opinions of what should be part of
> > Kafka etc. I agree with Suresh, let's have a vote on if we actually want
> a
> > REST client in Kafka core, and then we can work out what it actually
> looks
> > like (personally I would like to reuse Confluents, great REST client if
> > they donate it to ASF)
> >
> > For me +1 on REST client, this is a fundamental feature that's missing
> from
> > Kafka.
> >
> > On Mon, Oct 24, 2016 at 9:22 PM, Jay Kreps <j...@confluent.io> wrote:
> >
> > > Hey Suresh,
> > >
> > > I think we agree that REST apis are useful. I don't think we agree that
> > > they need to be part of the Kafka project. We've had this discussion a
> > > dozen odd times for different protocols---AMQP, REST, MQTT. Going back
> > the
> > > last five years we've always rejected these. That doesn't mean they
> > aren't
> > > useful, I think having these as separate is fine, they don't have any
> > > complex interaction with Kafka, they just use the vanilla APIs like any
> > of
> > > the dozens of things on the ecosystem page.
> > >
> > > In terms of how they're developed. I think there are two discussions
> > here:
> > > 1. Separate project or not
> > > 2. Standalone Apache project or github
> > >
> > > The first of those I just talked about---I think this makes sense as an
> > > independent project.
> > >
> > > For the second of these, I actually don't think that spawning off a
> bunch
> > > of itty-bitty independent Apache projects is a good thing. I think you
> > can
> > > see this in the Hadoop ecosystem where the parts kind of all evolve off
> > in
> > > different directions and don't really work together as a whole. I think
> > > that makes sense for deep infrastructure projects, but not for these
> > little
> > > helper projects. I think a hub and spoke model where you have a central
> > > project (Kafka) and then a bunch of helper tools that strive to fit in
> > with
> > > it (in terms of config, monitoring, apis, and every other conventions)
> is
> > > actually much better. In any case there are already so many of these
> > tools
> > > (we capture maybe 20% of them on the ecosystem page), made by so many
> > > people, and virtually all developed in this style, it is a bit late to
> > put
> > > the cat back in the bag.
> > >
> > > Perhaps it's just a difference in background. For my part I had many
> > years
> > > successfully running github-style projects, and i think that model can
> > work
> > > quite well for small things. I do think it is important for these
> > projects
> > > to clarify governance, which we should absolutely do, but realistically
> > it
> > > is a pretty simple tool, there isn't a huge governance challenge for
> > > something like this because its scope is so limited ("take http
> requests,
> > > make Kafka requests").
> > >
> > > More specifically, I don't think there is an actual problem being
> solved
> > > here. I haven't heard any complaint about direction or patches not
> > getting
> > > added. The only complaint I've heard is missing features where the
> normal
> > > "patches accepted" rule applies. I would urge people to actually get
> > > involved in contribution here. In the future if there is a situation
> > where
> > > people don't like the direction of a given tool, they can fork it and
> > > either turn it into an independent Apache project or develop it
> > > independently, trying to do that preemptively seems a bit hostile.
> > >
> > > -Jay
> > >
> > > On Mon, Oct 24, 2016 at 12:32 PM, Suresh Srinivas <
> > sur...@hortonworks.com>
> > > wrote:
> > >
> > > > I am dividing this discussion into two parts:
> > > > 1. REST APIs as core Apache Kafka capability
> > > > This should be a core Kafka functionality. Same view has been
> reflected
> > > by
> > > > others (users and developers) as well. While we can debate whether
> > other
> > > > capabilities are core Kafka (Streams, Connect), it would be good
> > separate
> > > > that out and to keep this discussion focussed on REST APIs as
> proposed
> > by
> > > > this KIP. If there is ambivalence about the need for this in core
> > kafka,
> > > > we could have voting in the mailing list.
> > > >
> > > > Can we get an agreement on this? I am +1 on REST API in Apache Kafka.
> > > >
> > > > 2. Community where Kafka REST APIs need to be collaborated on
> > > > There is already a GitHub project that caters to Kafka REST APIs.
> But a
> > > > company owned Github is less than ideal for collaboration due to lack
> > of
> > > > governance, open community and roadmap. This is reflected by many
> > people
> > > > interested in this functionality and still not contributing to and
> > > > adopting the code base in the GitHub. I think trying overlay the ASF
> > > > governance model on GitHub project, which is what the need is, seems
> > > > unnecessary, if the code can be part of Apache Kafka.
> > > >
> > > > The question is, would Confluent be okay with licensing/contributing
> > the
> > > > code to Kafka project (assuming either Confluent or another
> contributor
> > > > can work on it)? I see clear intent in collaborating with others on
> > REST
> > > > APIs from confluent. Why not do it in Kafka project under ASF? This
> > takes
> > > > away all the barrier to collaboration? Alternatively, if Confluent is
> > not
> > > > willing to contribute the code from GitHub, would anyone veto
> building
> > a
> > > > separate REST API functionality in ASF Kafka? There are enough people
> > who
> > > > want to work on this and maintain it.
> > > >
> > > > Regards,
> > > > Suresh
> > > >
> > > >
> > > >
> > > > On 10/21/16, 9:41 PM, "Harsha Chintalapani" <ka...@harsha.io> wrote:
> > > >
> > > > >Sriram,
> > > > > "Can the streaming platform exist without stream processing? -
> No.
> > > > >Processing stream data again is a core part of streaming platform."
> > > > >
> > > > >Yes, it can. There are no.of Stream processing frameworks out there,
> > and
> > > > >they all have integration into Kafka.
> > > > >It doesn't need to be developed within Kafka.
> > > > >
> > > > >
> > > > >"Can the platform exist without the rest proxy? - Yes. The proxy
> does
> > > not
> > > > >complete the platform vision in anyway. It is just a good to have
> tool
> > > > >that
> > > > >might be required by quite a few users and there is an active
> project
> > > that
> > > > >works on this - https://github.com/confluentinc/kafka-rest";
> > > > >
> > > > >The rest proxy is as important as any API. The vision that shown
> here
> > > > >http://kafka.apache.org/intro
> > > > >require users to write the producers and consumers to get their data
> > > into
> > > > >and out of Kafka, without which having Streams or Connect won't help
> > > > >anyone.
> > > > >The rest proxy makes easier for users get their data into Kafka.
> > > > >Adding the rest proxy to the project doesn't invalidate the current
> > > > >vision,
> > > > >it only strengthens it.
> > > > >
> > > > >Thanks,
> > > > >Harsha
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >On Fri, Oct 21, 2016 at 2:31 PM Sriram Subramanian <
> r...@confluent.io>
> > > > >wrote:
> > > > >
> > > > >FWIW, Apache Kafka has evolved a lot from where it started. It did
> > start
> > > > >as
> > > > >a messaging system. Over time we realized that that the vision for
> > Kafka
> > > > >is
> > > > >to build a streaming platform and not just a messaging system. You
> can
> > > > >take
> > > > >a look at the site for more description about what comprises the
> > > streaming
> > > > >platform http://kafka.apache.org/ and http://kafka.apache.org/intro
> .
> > > > >
> > > > >Can the streaming platform exist without Connect? - No. Data
> > integration
> > > > >is
> > > > >fundamental to building an end to end platform
> > > > >
> > > > >Can the streaming platform exist without stream processing? - No.
> > > > >Processing stream data again is a core part of streaming platform.
> > > > >
> > > > >Can the streaming platform exist without clients? - We at least need
> > one
> > > > >client library to complete the platform. Our Java clients help us to
> > > > >complete the platform story. The rest of the clients are built and
> > > > >maintained outside the project.
> > > > >
> > > > >Can the platform exist without the rest proxy? - Yes. The proxy does
> > not
> > > > >complete the platform vision in anyway. It is just a good to have
> tool
> > > > >that
> > > > >might be required by quite a few users and there is an active
> project
> > > that
> > > > >works on this - https://github.com/confluentinc/kafka-rest
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis
> > > > ><nso...@linkedin.com.invalid>
> > > > >wrote:
> > > > >
> > > > >> Are you saying Kafka REST is subjective but Kafka Streams and
> Kafka
> > > > >Connect
> > > > >> are not subjective?
> > > > >>
> > > > >> > "there are likely places that can live without a rest proxy"
> > > > >>
> > > > >> There are also places that can live without Kafka Streams and
> Kafka
> > > > >> Connect.
> > > > >>
> > > > >> Nacho
> > > > >>
> > > > >> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao <j...@confluent.io>
> wrote:
> > > > >>
> > > > >> > At the high level, I think ideally it makes sense to add a
> > component
> > > > >>to
> > > > >> > Apache Kafka if (1) it's widely needed and (2) it needs tight
> > > > >integration
> > > > >> > with Kafka core. For Kafka Stream, we do expect stream
> processing
> > > will
> > > > >be
> > > > >> > used widely in the future. Implementation wise, Kafka Stream
> only
> > > > >> supports
> > > > >> > getting data from Kafka and leverages quite a few of the core
> > > > >> > functionalities in Kafka core. For example, it uses customized
> > > > >>rebalance
> > > > >> > callback in the consumer and uses the compacted topic heavily.
> So,
> > > > >having
> > > > >> > Kafka Stream in the same repo makes it easier for testing when
> > those
> > > > >core
> > > > >> > functionalities evolve over time. Kafka Connect is in the same
> > > > >situation.
> > > > >> >
> > > > >> > For rest proxy, whether it's widely used or not is going to be a
> > bit
> > > > >> > subjective. However, there are likely places that can live
> > without a
> > > > >rest
> > > > >> > proxy. The rest proxy is just a proxy for the regular clients
> and
> > > > >doesn't
> > > > >> > need to be tightly integrated with Kafka core. So, the case for
> > > > >including
> > > > >> > rest proxy in Apache Kafka is probably not as strong as Kafka
> > Stream
> > > > >>and
> > > > >> > Kafka Connect.
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Jun
> > > > >> >
> > > > >> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce
> > > > >><michael.pea...@ig.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > So from my reading essentially the first question needs to
> > > > >answered/and
> > > > >> > > voted on is:
> > > > >> > >
> > > > >> > > Is Apache Kafka Community only about the Core or does the
> apache
> > > > >> > community
> > > > >> > > also support some subprojects (and just we need some better
> way
> > to
> > > > >> manage
> > > > >> > > this)
> > > > >> > >
> > > > >> > > If vote for Core only wins, then the following should be
> > removed:
> > > > >> > > Kafka Connect
> > > > >> > > Kafka Stream
> > > > >> > >
> > > > >> > > If vote for Core only loses (aka we will support subprojects)
> > > then:
> > > > >> > > We should look to add Kafka Rest
> > > > >> > >
> > > > >> > > And we should look to see how we can manage better govern and
> > > manage
> > > > >> > > submodules.
> > > > >> > >
> > > > >> > > A good example which id propose here is how some other
> > communities
> > > > >>in
> > > > >> > > Apache do this.
> > > > >> > >
> > > > >> > > Each Module has a Module Management Committee(MMC), this is
> like
> > > > >almost
> > > > >> > > the PMC but at a per module basis.
> > > > >> > >
> > > > >> > > This MMC should essentially hold the binding votes for that
> > > module.
> > > > >> > > The MMC should be made up of a single representative from each
> > > > >> > > organisation (so no single organisation can fully veto the
> > > > >>community
> > > > >> it
> > > > >> > > has to a genuine consenus)
> > > > >> > > The MMC requires at least 3 members (so there cant be a tied
> > vote
> > > on
> > > > >2)
> > > > >> > > For a new Module to be added a MMC committee should be sought
> > > > >> > > A new Module is only capable of being added if the above
> > > > >>requirements
> > > > >> can
> > > > >> > > be met (e.g. 3 people wishing to step up, from 3
> organisations)
> > so
> > > > >that
> > > > >> > > only actively support modules would be added
> > > > >> > >
> > > > >> > > The PMC reviews each module every 6months or Year. If MMC is
> > > > >>inactive,
> > > > >> a
> > > > >> > > vote/call to find replacements if raised, if none are
> > forthcoming
> > > > >> > dropping
> > > > >> > > the MMC to less than 3 then the module moves to "the attic"
> > (very
> > > > >>much
> > > > >> > like
> > > > >> > > apache attic but a little more aggressively)
> > > > >> > >
> > > > >> > > This way the PMC does not need to micro manage every module
> > > > >> > > We only add modules where some amount of active support and
> > > > >maintenance
> > > > >> > > and use is provided by the community
> > > > >> > > We have an automatic way to retire old or inactive projects.
> > > > >> > >
> > > > >> > > Thoughts?
> > > > >> > > Mike
> > > > >> > >
> > > > >> > >
> > > > >> > > ________________________________________
> > > > >> > > From: Harsha Ch <harsha...@gmail.com>
> > > > >> > > Sent: Thursday, October 20, 2016 10:26 PM
> > > > >> > > To: dev@kafka.apache.org
> > > > >> > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > >> > >
> > > > >> > > Jay,
> > > > >> > > REST API is something every user is in need of. If the
> > > > >>argument
> > > > >> is
> > > > >> > to
> > > > >> > > clone and write your API, this will do a disservice to the
> > users
> > > as
> > > > >> they
> > > > >> > > now have to choose one vs. others instead of keeping one API
> > that
> > > is
> > > > >> > > supported in Kafka community.
> > > > >> > >
> > > > >> > > "Pre-emptively re-creating another
> > > > >> > > REST layer when it seems like we all quite agree on what needs
> > to
> > > be
> > > > >> done
> > > > >> > > and we have an existing code base for HTTP/Kafka access that
> is
> > > > >heavily
> > > > >> > > used in production seems quite silly."
> > > > >> > >
> > > > >> > > Exactly our point. Why can't we develop this in Apache
> > > Kafka
> > > > >> > > community? Instead of us open sourcing another GitHub project
> > and
> > > > >> > creating
> > > > >> > > a divide in users and another version of API. Let's build this
> > in
> > > > >Kafka
> > > > >> > > Community and use the governance model that is proven to
> provide
> > > > >vendor
> > > > >> > > free user driven consensus features. The argument that is
> adding
> > > > >>this
> > > > >> > REST
> > > > >> > > server to Kafka will affect the agility of the project doesn't
> > mak
> > > > >> sense.
> > > > >> > >
> > > > >> > > It looks like your argument is either we develop all these
> small
> > > > >>tools
> > > > >> or
> > > > >> > > none at all. We as a community need to look at supporting
> > critical
> > > > >> > > tools/API. Instead of dividing this project into individual
> > > external
> > > > >> > > communities. We should build this as part of Kafka which best
> > > serves
> > > > >> the
> > > > >> > > needs of users.
> > > > >> > > The Streams and Connect projects that were pushed into
> > > Kafka
> > > > >> > could
> > > > >> > > have been left in their own Github projects based on your
> > > arguments.
> > > > >> What
> > > > >> > > about the REST API is so different that such that it should
> stay
> > > out
> > > > >of
> > > > >> > the
> > > > >> > > Kafka project? From my experience, more users are asking for
> the
> > > > >>REST
> > > > >> > API.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Harsha
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Wed, Oct 12, 2016 at 8:03 AM Jay Kreps <j...@confluent.io>
> > > wrote:
> > > > >> > >
> > > > >> > > > I think the questions around governance make sense, I think
> we
> > > > >should
> > > > >> > > > really clarify that to make the process more clear so it can
> > be
> > > > >fully
> > > > >> > > > inclusive.
> > > > >> > > >
> > > > >> > > > The idea that we should not collaborate on what is there
> now,
> > > > >though,
> > > > >> > > > because in the future we might disagree about direction does
> > not
> > > > >> really
> > > > >> > > > make sense to me. If in the future we disagree, that is the
> > > beauty
> > > > >of
> > > > >> > > open
> > > > >> > > > source, you can always fork off a copy of the code and start
> > an
> > > > >> > > independent
> > > > >> > > > project either in Apache or elsewhere. Pre-emptively
> > re-creating
> > > > >> > another
> > > > >> > > > REST layer when it seems like we all quite agree on what
> needs
> > > to
> > > > >>be
> > > > >> > done
> > > > >> > > > and we have an existing code base for HTTP/kafka access that
> > is
> > > > >> heavily
> > > > >> > > > used in production seems quite silly.
> > > > >> > > >
> > > > >> > > > Let me give some background on how I at least think about
> > these
> > > > >> things.
> > > > >> > > > I've participated in open source projects out of LinkedIn
> via
> > > > >>github
> > > > >> as
> > > > >> > > > well as via the ASF. I don't think there is a "right" answer
> > to
> > > > >>how
> > > > >> to
> > > > >> > do
> > > > >> > > > these but rather some tradeoffs. We thought about this
> quite a
> > > lot
> > > > >in
> > > > >> > the
> > > > >> > > > context of Kafka based on the experience with the Hadoop
> > > ecosystem
> > > > >as
> > > > >> > > well
> > > > >> > > > as from other open source communities.
> > > > >> > > >
> > > > >> > > > There is a rich ecosystem around Kafka. Many of the projects
> > are
> > > > >> quite
> > > > >> > > > small--single clients or tools that do a single thing
> > well--and
> > > > >> almost
> > > > >> > > none
> > > > >> > > > of them are top level apache projects. I don't think trying
> to
> > > > >>force
> > > > >> > each
> > > > >> > > > of these to turn into independent Apache projects is
> > necessarily
> > > > >>the
> > > > >> > best
> > > > >> > > > thing for the ecosystem.
> > > > >> > > >
> > > > >> > > > My observation of how this can go wrong is really what I
> think
> > > has
> > > > >> > > happened
> > > > >> > > > to the Hadoop ecosystem. There you see quite a zoo of
> projects
> > > > >>which
> > > > >> > all
> > > > >> > > > drift apart and don't quite work together well. Coordinating
> > > even
> > > > >> > simple
> > > > >> > > > changes and standardization across these is exceptionally
> > > > >>difficult.
> > > > >> > The
> > > > >> > > > result is a bit of a mess for users--the pieces just don't
> > > really
> > > > >> come
> > > > >> > > > together very well. This makes sense for independent
> > > > >>infrastructure
> > > > >> > > systems
> > > > >> > > > (Kudu vs HDFS) but I'm not at all convinced that doing this
> > for
> > > > >every
> > > > >> > > > little tool or helper library has lead to a desirable
> state. I
> > > > >>think
> > > > >> > the
> > > > >> > > > mode of operating where the Hadoop vendors spawn off a few
> new
> > > > >Apache
> > > > >> > > > projects for each new product initiative, especially since
> > often
> > > > >that
> > > > >> > > > project is only valued by that vendor (and the other vendor
> > has
> > > a
> > > > >> > > different
> > > > >> > > > competing Apache project) doesn't necessarily do a better
> job
> > at
> > > > >> > > producing
> > > > >> > > > high quality communities or high quality software.
> > > > >> > > >
> > > > >> > > > These tools/connects/clients/proxies and other integration
> > > pieces
> > > > >can
> > > > >> > > take
> > > > >> > > > many forms, but my take of what makes one of these things
> good
> > > is
> > > > >> that
> > > > >> > it
> > > > >> > > > remains simple, does its one thing well, and cleaves as
> > closely
> > > as
> > > > >> > > possible
> > > > >> > > > to the conventions for Kafka itself--i.e. doesn't invent new
> > > ways
> > > > >>of
> > > > >> > > > monitoring, configuring, etc. For the tools we've
> contributed
> > > > >>we've
> > > > >> > tried
> > > > >> > > > really hard to make them consistent with Kafka as well as
> with
> > > > >>each
> > > > >> > other
> > > > >> > > > in how testing, configuration, monitoring, etc works.
> > > > >> > > >
> > > > >> > > > I think what Apache does superbly well is create a community
> > for
> > > > >> > > managing a
> > > > >> > > > large infrastructure layer like Kafka in a vendor
> independent
> > > way.
> > > > >> > What I
> > > > >> > > > think is less successful is attempting to form full and
> > > > >>independent
> > > > >> > > apache
> > > > >> > > > communities around very simple single purpose tools,
> > especially
> > > if
> > > > >> you
> > > > >> > > hope
> > > > >> > > > for these to come together into a cohesive toolset across
> > > multiple
> > > > >> such
> > > > >> > > > tools. Much of what Apache does--create a collective
> decision
> > > > >>making
> > > > >> > > > process for resolving disagreement, help to trademark and
> > > protect
> > > > >the
> > > > >> > > marks
> > > > >> > > > of the project, etc just isn't that relevant for simple
> > > > >> single-purpose
> > > > >> > > > tools.
> > > > >> > > >
> > > > >> > > > So my take is there are a couple of options:
> > > > >> > > >
> > > > >> > > > 1. We can try to put all the small tools into the Apache
> > > > >>Project.
> > > > >> I
> > > > >> > > > think this is not the right approach as there is simply
> too
> > > > >>many
> > > > >> of
> > > > >> > > > them,
> > > > >> > > > many in different languages, serving different protocols,
> > > > >> > integrating
> > > > >> > > > with
> > > > >> > > > particular systems, and a single community can't
> > effectively
> > > > >> > maintain
> > > > >> > > > them
> > > > >> > > > all. Doing this would significantly slow the progress of
> > the
> > > > >Kafka
> > > > >> > > > project.
> > > > >> > > > As a protocol for messaging, I don't really see a case
> for
> > > > >> including
> > > > >> > > > REST
> > > > >> > > > but not MQTT or AMQP which are technically much better
> > suited
> > > > >>to
> > > > >> > > > messaging
> > > > >> > > > and both are widely used for that.
> > > > >> > > > 2. We can treat ecosystem projects that aren't top level
> > > Apache
> > > > >> > > projects
> > > > >> > > > as invalid and try to recreate them all as Apache
> projects.
> > > > >> > Honestly,
> > > > >> > > > though, if you go to the Kafka ecosystem page virtually
> > none
> > > of
> > > > >> the
> > > > >> > > most
> > > > >> > > > popular add-ons to Kafka are Apache projects. The most
> > > > >>successful
> > > > >> > > > things in
> > > > >> > > > the Kafka ecosystem such as Yahoo Manager, librdkafka, a
> > > number
> > > > >of
> > > > >> > > other
> > > > >> > > > clients, as well as the existing REST layer have
> succeeded
> > at
> > > > >> > > developing
> > > > >> > > > communities that actively contribute and use these pieces
> > > and I
> > > > >> > don't
> > > > >> > > > know
> > > > >> > > > that that is a bad thing unless that community proves to
> be
> > > > >> > > uninclusive,
> > > > >> > > > unresponsive, or goes in a bad technical direction--and
> > those
> > > > >>are
> > > > >> > > > failure
> > > > >> > > > modes that all open source efforts face.
> > > > >> > > > 3. We can do what I think makes the most sense and try to
> > > work
> > > > >> with
> > > > >> > > the
> > > > >> > > > projects that exist in the ecosystem and if the project
> > > doesn't
> > > > >> > have a
> > > > >> > > > responsive community or wants to go in a different
> > direction
> > > > >>fork
> > > > >> or
> > > > >> > > > recreate that work.
> > > > >> > > >
> > > > >> > > > Of course any person can choose whatever of these options
> they
> > > > >>want.
> > > > >> > But
> > > > >> > > > from my point of view, option (3) has been the path of the
> > > > >>community
> > > > >> so
> > > > >> > > far
> > > > >> > > > and I think it has been quite successful.
> > > > >> > > >
> > > > >> > > > -Jay
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Tue, Oct 11, 2016 at 10:33 PM, Harsha Chintalapani <
> > > > >> ka...@harsha.io
> > > > >> > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Neha,
> > > > >> > > > > "But I haven't seen any community emails or patches being
> > > > >submitted
> > > > >> > by
> > > > >> > > > you
> > > > >> > > > > guys, so I'm wondering why you are concerned about whether
> > the
> > > > >> > > community
> > > > >> > > > is
> > > > >> > > > > open to accepting patches or not."
> > > > >> > > > >
> > > > >> > > > > I think you are talking about contributing patches to this
> > > > >> repository
> > > > >> > > > > right? https://github.com/confluentinc/kafka-rest . All I
> > am
> > > > >> saying
> > > > >> > > > > the guidelines/governance model is not clear on the
> project
> > > and
> > > > >>I
> > > > >> > guess
> > > > >> > > > its
> > > > >> > > > > driven by opening a github issue request. Its the
> > repository
> > > > >owned
> > > > >> > by
> > > > >> > > > > confluent and as much I appreciate that the features we
> > > > >>mentioned
> > > > >> are
> > > > >> > > in
> > > > >> > > > > the roadmap and welcoming us to contribute to the project.
> > It
> > > > >> doesn't
> > > > >> > > > > gurantee what we want to add in the furture will be in
> your
> > > > >> roadmap.
> > > > >> > > > >
> > > > >> > > > > Hence the reason having it part of Kafka community will
> > help a
> > > > >>lot
> > > > >> as
> > > > >> > > > other
> > > > >> > > > > users can participate in the discussions. We are happy to
> > > drive
> > > > >> any
> > > > >> > > > > feature additions through KIPs which gives everyone a
> chance
> > > to
> > > > >> > > > participate
> > > > >> > > > > and add to the discussions.
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Harsha
> > > > >> > > > >
> > > > >> > > > > On Fri, Oct 7, 2016 at 11:52 PM Michael Pearce <
> > > > >> > michael.pea...@ig.com>
> > > > >> > > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > +1
> > > > >> > > > > >
> > > > >> > > > > > I agree on the governance comments whole heartedly.
> > > > >> > > > > >
> > > > >> > > > > > Also i agree about the contribution comments made
> earlier
> > in
> > > > >>the
> > > > >> > > > thread,
> > > > >> > > > > i
> > > > >> > > > > > personally am less likely to spend any of mine, or give
> > > > >>project
> > > > >> > time
> > > > >> > > > > within
> > > > >> > > > > > my internal projects to developers contributing to
> another
> > > > >> > commercial
> > > > >> > > > > > companies project even so technically open source, as
> then
> > > > >>there
> > > > >> is
> > > > >> > > > that
> > > > >> > > > > > commercial companies interest will always prevail and
> > > > >essentially
> > > > >> > can
> > > > >> > > > > > always have a final vote where disagreement. Im sure
> they
> > > > >>never
> > > > >> > > intend
> > > > >> > > > > to,
> > > > >> > > > > > but there is that true reality. This is why we have
> > > community
> > > > >> open
> > > > >> > > > source
> > > > >> > > > > > projects.
> > > > >> > > > > >
> > > > >> > > > > > I can find many different implementations now of a rest
> > > > >>endpoint
> > > > >> on
> > > > >> > > > > > GitHub, BitBucket etc. Each one has their benefits and
> > > > >> > disadvantages
> > > > >> > > in
> > > > >> > > > > > their implementation. By making / providing one this
> would
> > > > >>bring
> > > > >> > > > together
> > > > >> > > > > > these solutions, unifying those developers and also
> > bringing
> > > > >>the
> > > > >> > best
> > > > >> > > > of
> > > > >> > > > > > all.
> > > > >> > > > > >
> > > > >> > > > > > I understand the concern on the community burden
> > > > >> adding/supporting
> > > > >> > > more
> > > > >> > > > > > surface area for every client. But something like REST
> is
> > > > >> universal
> > > > >> > > and
> > > > >> > > > > > worthy to be owned by the community.
> > > > >> > > > > >
> > > > >> > > > > > Mike
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > ________________________________________
> > > > >> > > > > > From: Andrew Schofield <andrew_schofield_jira@
> outlook.com
> > >
> > > > >> > > > > > Sent: Saturday, October 8, 2016 1:19 AM
> > > > >> > > > > > To: dev@kafka.apache.org
> > > > >> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > >> > > > > >
> > > > >> > > > > > There's a massive difference between the governance of
> > Kafka
> > > > >>and
> > > > >> > the
> > > > >> > > > > > governance of the REST proxy.
> > > > >> > > > > >
> > > > >> > > > > > In Kafka, there is a broad community of people
> > contributing
> > > > >their
> > > > >> > > > > opinions
> > > > >> > > > > > about future enhancements in the form of KIPs. There's
> > some
> > > > >> really
> > > > >> > > deep
> > > > >> > > > > > consideration that goes into some of the trickier KIPs.
> > > There
> > > > >are
> > > > >> > > > people
> > > > >> > > > > > outside Confluent deeply knowledgeable in Kafka and
> > > building
> > > > >the
> > > > >> > > > > > reputations to become committers. I get the impression
> > that
> > > > >>the
> > > > >> > > roadmap
> > > > >> > > > > of
> > > > >> > > > > > Kafka is not really community-owned (what's the big
> > feature
> > > > >>for
> > > > >> > Kafka
> > > > >> > > > > 0.11,
> > > > >> > > > > > for example), but the conveyor belt of smaller features
> in
> > > the
> > > > >> form
> > > > >> > > of
> > > > >> > > > > KIPs
> > > > >> > > > > > works nicely. It's a good example of open-source
> working
> > > > >>well.
> > > > >> > > > > >
> > > > >> > > > > > The equivalent for the REST proxy is basically issues on
> > > > >>GitHub.
> > > > >> > The
> > > > >> > > > > > roadmap is less clear. There's not really a community
> > > properly
> > > > >> > > engaged
> > > > >> > > > in
> > > > >> > > > > > the way that there is with Kafka. So, you could say that
> > > it's
> > > > >> clear
> > > > >> > > > that
> > > > >> > > > > > fewer people are interested, but I think the whole
> > > governance
> > > > >> > thing
> > > > >> > > > is a
> > > > >> > > > > > big barrier to engagement. And it's looking like it's
> > > getting
> > > > >out
> > > > >> > of
> > > > >> > > > > date.
> > > > >> > > > > >
> > > > >> > > > > > In technical terms, I can think of two big improvements
> to
> > > the
> > > > >> REST
> > > > >> > > > > proxy.
> > > > >> > > > > > First, it needs to use the new consumer API so that it's
> > > > >possible
> > > > >> > to
> > > > >> > > > > secure
> > > > >> > > > > > connections between the REST proxy and Kafka. I don't
> care
> > > too
> > > > >> much
> > > > >> > > > which
> > > > >> > > > > > method calls it uses actually uses to consume messages,
> > > but I
> > > > >do
> > > > >> > > care
> > > > >> > > > > that
> > > > >> > > > > > I cannot secure connections because of network security
> > > rules.
> > > > >> > > Second,
> > > > >> > > > > > there's an affinity between a consumer and the instance
> of
> > > the
> > > > >> REST
> > > > >> > > > proxy
> > > > >> > > > > > to which it first connected. Kafka itself avoids this
> kind
> > > of
> > > > >> > > affinity
> > > > >> > > > > for
> > > > >> > > > > > good reason, and in the name of availability the REST
> > proxy
> > > > >> should
> > > > >> > > too.
> > > > >> > > > > > These are natural KIPs.
> > > > >> > > > > >
> > > > >> > > > > > I think it would be good to have the code for the REST
> > proxy
> > > > >> > > > contributed
> > > > >> > > > > > to Apache Kafka so that it would be able to be developed
> > in
> > > > >>the
> > > > >> > same
> > > > >> > > > way.
> > > > >> > > > > >
> > > > >> > > > > > Andrew Schofield
> > > > >> > > > > >
> > > > >> > > > > > From: Suresh Srinivas <sur...@hortonworks.com>
> > > > >> > > > > > Sent: 07 October 2016 22:41:52
> > > > >> > > > > > To: dev@kafka.apache.org
> > > > >> > > > > > Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
> > > > >> > > > > >
> > > > >> > > > > > ASF already gives us a clear framework and governance
> > model
> > > > >>for
> > > > >> > > > community
> > > > >> > > > > > development. This is already understood by the people
> > > > >> contributing
> > > > >> > to
> > > > >> > > > > > Apache Kafka project, and they are the same people who
> > want
> > > to
> > > > >> > > > contribute
> > > > >> > > > > > to the REST server capability as well. Everyone is in
> > > > >>agreement
> > > > >> on
> > > > >> > > the
> > > > >> > > > > > need for collaborating on this effort. So why not
> > contribute
> > > > >>the
> > > > >> > code
> > > > >> > > > to
> > > > >> > > > > > Apache Kafka. This will help avoid duplication of effort
> > and
> > > > >> forks
> > > > >> > > that
> > > > >> > > > > > may crop up, hugely benefitting the user community. This
> > > will
> > > > >> also
> > > > >> > > > avoid
> > > > >> > > > > > having to define a process similar to ASF on a GitHub
> > > project
> > > > >and
> > > > >> > > > instead
> > > > >> > > > > > there is a single community with clear understanding
> > > community
> > > > >> > > process
> > > > >> > > > as
> > > > >> > > > > > defined in ASF.
> > > > >> > > > > >
> > > > >> > > > > > As others have said, this is an important capability for
> > > > >>Apache
> > > > >> > > Kafka.
> > > > >> > > > It
> > > > >> > > > > > is worth maintaining this as a part of the project.
> > > > >> > > > > >
> > > > >> > > > > > Regards,
> > > > >> > > > > > Suresh
> > > > >> > > > > >
> > > > >> > > > > > On 10/6/16, 8:32 AM, "Ofir Manor" <
> ofir.ma...@equalum.io>
> > > > >>wrote:
> > > > >> > > > > >
> > > > >> > > > > > >I personally think it would be quite wasteful to
> > > re-implement
> > > > >> the
> > > > >> > > REST
> > > > >> > > > > > >gateway just because that an actively-maintained piece
> of
> > > > >> > > > > Apache-licensed
> > > > >> > > > > > >software is not governed directly by the Apache Kafka
> > > > >> community...
> > > > >> > > > While
> > > > >> > > > > > >kafka-rest repo is owned by Confluent, the contributors
> > > > >> including
> > > > >> > > the
> > > > >> > > > > main
> > > > >> > > > > > >one are also part of the Apache Kafka community, so
> > there
> > > > >>is a
> > > > >> > > chance
> > > > >> > > > > to
> > > > >> > > > > > >work this out.
> > > > >> > > > > > >
> > > > >> > > > > > >However, there are two valid concerns here that could
> be
> > > > >> > addressed,
> > > > >> > > > > around
> > > > >> > > > > > >community and accessibility:
> > > > >> > > > > > >>> What we are worried about is a project
> > > > >> > > > > > >>> that's not maintained in a community. So the process
> > of
> > > > >> > accepting
> > > > >> > > > > > >>>patches
> > > > >> > > > > > >>> and priorities is not clear, and it's not developed
> in
> > > > >Apache
> > > > >> > > > > > >>>community.
> > > > >> > > > > > >>> Not only that, existing REST API project doesn't
> > support
> > > > >>new
> > > > >> > > client
> > > > >> > > > > API
> > > > >> > > > > > >and
> > > > >> > > > > > >>> hence there is no security support either.
> > > > >> > > > > > >
> > > > >> > > > > > >This might be easy to fix. Maybe Confluent / kafka-rest
> > > > >> community
> > > > >> > > can
> > > > >> > > > > > >clarify that - what is their contribution policy, dev
> > > style,
> > > > >> > roadmap
> > > > >> > > > > etc.
> > > > >> > > > > > >If they want, they can make an effort to encourage
> > > > >participation
> > > > >> > > from
> > > > >> > > > > > >people outside Confluent (easily accept contributions,
> > > invite
> > > > >> > > external
> > > > >> > > > > > >commiters or have open dev process similar to Apache
> > Kafka
> > > > >etc),
> > > > >> > as
> > > > >> > > > > there
> > > > >> > > > > > >is definitely seems to be some interest on the list.
> That
> > > > >>might
> > > > >> > > clear
> > > > >> > > > > the
> > > > >> > > > > > >community concern and help kafka-rest project (but that
> > is
> > > a
> > > > >> > > > calculation
> > > > >> > > > > > >Confluent will have to make).
> > > > >> > > > > > >
> > > > >> > > > > > >The other, independent, concern is that REST is
> something
> > > > >>that
> > > > >> is
> > > > >> > > > > expected
> > > > >> > > > > > >to be available out of the box with Kafka. I personally
> > > don't
> > > > >> feel
> > > > >> > > > > > >strongly
> > > > >> > > > > > >about it (better use proper, efficient APIs from day
> > one),
> > > > >> though
> > > > >> > it
> > > > >> > > > is
> > > > >> > > > > > >definitely way smaller than adding a stream processing
> > > engine
> > > > >to
> > > > >> > the
> > > > >> > > > > > >project :)
> > > > >> > > > > > >Again,the kafka-rest "community" could take steps to
> make
> > > it
> > > > >> even
> > > > >> > > > easier
> > > > >> > > > > > >to
> > > > >> > > > > > >install, configure and run kafka-rest for new users on
> > > > >>vanilla
> > > > >> > > Apache
> > > > >> > > > > > >Kafka
> > > > >> > > > > > >(outside the Confluent platform), if they wish that (or
> > > > >>welcome
> > > > >> > > > > > >contributions to that end), but that is up to them.
> > > > >> > > > > > >Finally, if after the above steps were taken there
> would
> > > > >>still
> > > > >a
> > > > >> > > > strong
> > > > >> > > > > > >desire to include a great rest gateway with Apache
> > Kafka, I
> > > > >> assume
> > > > >> > > the
> > > > >> > > > > > >community could hypothetically fork the existing
> > kafka-rest
> > > > >into
> > > > >> > an
> > > > >> > > > > Apache
> > > > >> > > > > > >Kafka subproject and maintain it "within Apache"
> instead
> > of
> > > > >> > > > implementing
> > > > >> > > > > > >it
> > > > >> > > > > > >from scratch (though I'm not a lawyer etc) - but I
> cannot
> > > > >> imagine
> > > > >> > it
> > > > >> > > > > > >happen
> > > > >> > > > > > >without Confluent blessing, and I think that is likely
> > much
> > > > >less
> > > > >> > > > optimal
> > > > >> > > > > > >(pulling in other Confluent / Apache licensed
> > dependencies)
> > > > >than
> > > > >> > > > having
> > > > >> > > > > a
> > > > >> > > > > > >separate external community around kafka-rest.
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >Just my two cents,
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > >Ofir Manor
> > > > >> > > > > > >
> > > > >> > > > > > >Co-Founder & CTO | Equalum
> > > > >> > > > > > >
> > > > >> > > > > > >Mobile: +972-54-7801286 <+972%2054-780-1286>
> > > > ><+972%2054-780-1286>
> > > > >> > <+972%2054-780-1286> |
> > > > >> > > > Email:
> > > > >> > > > > > ofir.ma...@equalum.io
> > > > >> > > > > > >
> > > > >> > > > > > >On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani <
> > > > >> > > ka...@harsha.io
> > > > >> > > > >
> > > > >> > > > > > >wrote:
> > > > >> > > > > > >
> > > > >> > > > > > >> Neha & Jay,
> > > > >> > > > > > >> We did look at the open source
> > > > >>alternatives.
> > > > >> > Our
> > > > >> > > > > > >>concern
> > > > >> > > > > > >> is what's the patch acceptance and adding features/
> > > > >>bug-fixes
> > > > >> to
> > > > >> > > the
> > > > >> > > > > > >> existing project under a Github (although it's
> licensed
> > > > >>under
> > > > >> > > Apache
> > > > >> > > > > > >>2.0).
> > > > >> > > > > > >> It would be great if that project made available
> under
> > > > >>Apache
> > > > >> > and
> > > > >> > > > > > >>driven by
> > > > >> > > > > > >> the community. Adding to the above, not all Kafka
> > users
> > > > >>are
> > > > >> > > > > interested
> > > > >> > > > > > >>in
> > > > >> > > > > > >> using the Java client API, they would like to have
> > simple
> > > > >REST
> > > > >> > API
> > > > >> > > > > where
> > > > >> > > > > > >> they can code against using any language. I do
> believe
> > > this
> > > > >> adds
> > > > >> > > > value
> > > > >> > > > > > >>to
> > > > >> > > > > > >> Apache Kafka in itself.
> > > > >> > > > > > >>
> > > > >> > > > > > >> "For 1, I don't think there is value in giving in to
> > the
> > > > >>NIH
> > > > >> > > > syndrome
> > > > >> > > > > > >>and
> > > > >> > > > > > >> reinventing the wheel. What I'm looking for is a
> > detailed
> > > > >> > > comparison
> > > > >> > > > > of
> > > > >> > > > > > >>the
> > > > >> > > > > > >> gaps and why those can't be improved in the REST
> proxy
> > > that
> > > > >> > > already
> > > > >> > > > > > >>exists
> > > > >> > > > > > >> and is actively maintained."
> > > > >> > > > > > >>
> > > > >> > > > > > >> We are not looking at this as NIH. What we are
> worried
> > > > >>about
> > > > >> > is a
> > > > >> > > > > > >>project
> > > > >> > > > > > >> that's not maintained in a community. So the process
> of
> > > > >> > accepting
> > > > >> > > > > > >>patches
> > > > >> > > > > > >> and priorities is not clear, and it's not developed
> in
> > > > >>Apache
> > > > >> > > > > community.
> > > > >> > > > > > >> Not only that, existing REST API project doesn't
> > support
> > > > >>new
> > > > >> > > client
> > > > >> > > > > API
> > > > >> > > > > > >>and
> > > > >> > > > > > >> hence there is no security support either.
> > > > >> > > > > > >> We don't know the timeline when that's made
> available.
> > We
> > > > >> would
> > > > >> > > like
> > > > >> > > > > to
> > > > >> > > > > > >>add
> > > > >> > > > > > >> admin functionality into the REST API. So the Roadmap
> > of
> > > > >>that
> > > > >> > > > project
> > > > >> > > > > is
> > > > >> > > > > > >> not driven by Apache.
> > > > >> > > > > > >>
> > > > >> > > > > > >>
> > > > >> > > > > > >> "This doesn't materially have an impact on expanding
> > the
> > > > >> > usability
> > > > >> > > > of
> > > > >> > > > > > >> Kafka. In my experience, REST proxy + Java clients
> > > only
> > > > >> cover
> > > > >> > > > ~50%
> > > > >> > > > > of
> > > > >> > > > > > >> all
> > > > >> > > > > > >> Kafka users, and maybe 10% of those are the ones
> who
> > > > >>will
> > > > >> use
> > > > >> > > the
> > > > >> > > > > > >>REST
> > > > >> > > > > > >> proxy. The remaining 50% are non-java client users
> > (C,
> > > > >> > python,
> > > > >> > > > go,
> > > > >> > > > > > >>node
> > > > >> > > > > > >> etc)."
> > > > >> > > > > > >>
> > > > >> > > > > > >> REST API is most often asked feature in my
> interactions
> > > > >>with
> > > > >> > Kafka
> > > > >> > > > > > >>users.
> > > > >> > > > > > >> In an organization, There will be independent teams
> who
> > > > >>will
> > > > >> > write
> > > > >> > > > > their
> > > > >> > > > > > >> Kafka clients using different language libraries
> > > available
> > > > >> > today,
> > > > >> > > > and
> > > > >> > > > > > >> there is no way to standardize this. Instead of
> > > supporting
> > > > >> > several
> > > > >> > > > > > >> different client libraries users will be interested
> in
> > > > >>using
> > > > >a
> > > > >> > > REST
> > > > >> > > > > API
> > > > >> > > > > > >> server. The need for a REST API will only increase as
> > > more
> > > > >and
> > > > >> > > more
> > > > >> > > > > > >>users
> > > > >> > > > > > >> start using Kafka.
> > > > >> > > > > > >>
> > > > >> > > > > > >> "More surface area means more work to keep things
> > > > >>consistent.
> > > > >> > > > Failure
> > > > >> > > > > > >> to do that has, in fact, hurt the user
> experience."
> > > > >> > > > > > >> Having myriad Kafka client GitHub projects that
> support
> > > > >> > different
> > > > >> > > > > > >>languages
> > > > >> > > > > > >> hurts the user experience and pushes burden to
> maintain
> > > > >>these
> > > > >> > > > > libraries.
> > > > >> > > > > > >> REST API is a simple code base that uses existing
> java
> > > > >>client
> > > > >> > > > > libraries
> > > > >> > > > > > >>to
> > > > >> > > > > > >> make life easier for the users.
> > > > >> > > > > > >>
> > > > >> > > > > > >> Thanks,
> > > > >> > > > > > >> Harsha
> > > > >> > > > > > >>
> > > > >> > > > > > >> On Sat, Oct 1, 2016 at 10:41 AM Neha Narkhede <
> > > > >> > n...@confluent.io>
> > > > >> > > > > > wrote:
> > > > >> > > > > > >>
> > > > >> > > > > > >> > Manikumar,
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > Thanks for sharing the proposal. I think there are
> 2
> > > > >>parts
> > > > >> to
> > > > >> > > this
> > > > >> > > > > > >> > discussion -
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > 1. Should we rewrite a REST proxy given that there
> > is a
> > > > >> > > > > > >>feature-complete,
> > > > >> > > > > > >> > open-source and actively maintained REST proxy in
> the
> > > > >> > community?
> > > > >> > > > > > >> > 2. Does adding a REST proxy to Apache Kafka make us
> > > more
> > > > >> agile
> > > > >> > > and
> > > > >> > > > > > >> maintain
> > > > >> > > > > > >> > the high-quality experience that Kafka users have
> > > today?
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > For 1, I don't think there is value in giving in to
> > the
> > > > >>NIH
> > > > >> > > > syndrome
> > > > >> > > > > > >>and
> > > > >> > > > > > >> > reinventing the wheel. What I'm looking for is a
> > > detailed
> > > > >> > > > comparison
> > > > >> > > > > > >>of
> > > > >> > > > > > >> the
> > > > >> > > > > > >> > gaps and why those can't be improved in the REST
> > proxy
> > > > >>that
> > > > >> > > > already
> > > > >> > > > > > >> exists
> > > > >> > > > > > >> > and is actively maintained. For example, we depend
> on
> > > > >> zkClient
> > > > >> > > and
> > > > >> > > > > > >>have
> > > > >> > > > > > >> > found as well as fixed several bugs by working
> > closely
> > > > >>with
> > > > >> > the
> > > > >> > > > > people
> > > > >> > > > > > >> who
> > > > >> > > > > > >> > maintain zkClient. This should be possible for REST
> > > proxy
> > > > >as
> > > > >> > > well,
> > > > >> > > > > > >>right?
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > For 2, I'd like us to review our history of
> expanding
> > > the
> > > > >> > > surface
> > > > >> > > > > > >>area to
> > > > >> > > > > > >> > add more clients in the past. Here is a summary -
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > - This doesn't materially have an impact on
> > > expanding
> > > > >the
> > > > >> > > > > > >>usability of
> > > > >> > > > > > >> > Kafka. In my experience, REST proxy + Java
> clients
> > > > >>only
> > > > >> > cover
> > > > >> > > > > ~50%
> > > > >> > > > > > >>of
> > > > >> > > > > > >> > all
> > > > >> > > > > > >> > Kafka users, and maybe 10% of those are the ones
> > who
> > > > >will
> > > > >> > use
> > > > >> > > > the
> > > > >> > > > > > >>REST
> > > > >> > > > > > >> > proxy. The remaining 50% are non-java client
> users
> > > (C,
> > > > >> > > python,
> > > > >> > > > > go,
> > > > >> > > > > > >> node
> > > > >> > > > > > >> > etc).
> > > > >> > > > > > >> > - People are a lot more excited about promising
> to
> > > > >> > contribute
> > > > >> > > > > while
> > > > >> > > > > > >> > adding the surface area but not on an ongoing
> > basis
> > > > >>down
> > > > >> > the
> > > > >> > > > > line.
> > > > >> > > > > > >> > - More surface area means more work to keep
> things
> > > > >> > > consistent.
> > > > >> > > > > > >>Failure
> > > > >> > > > > > >> > to do that has, in fact, hurt the user
> experience.
> > > > >> > > > > > >> > - More surface area hurts agility. We want to
> do a
> > > few
> > > > >> > things
> > > > >> > > > > > >>really
> > > > >> > > > > > >> > well as well as be agile to be able to build on
> > our
> > > > >>core
> > > > >> > > > > > >>competency.
> > > > >> > > > > > >> >
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > On Sat, Oct 1, 2016 at 5:38 AM Manikumar <
> > > > >> > > > manikumar.re...@gmail.com
> > > > >> > > > > >
> > > > >> > > > > > >> > wrote:
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > > Hi Jay,
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > Thanks for your reply.
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > I agree that we can not add all the clients/tools
> > > > >> available
> > > > >> > in
> > > > >> > > > > > >> ecosystem
> > > > >> > > > > > >> > > page to Kafka repo itself. But we feel REST
> > Interface
> > > > >>is
> > > > >> > > > different
> > > > >> > > > > > >>from
> > > > >> > > > > > >> > > other clients/tools. Since any language that can
> > work
> > > > >with
> > > > >> > > HTTP
> > > > >> > > > > can
> > > > >> > > > > > >> > > easily integrate with this interface, Having an
> > > > >"official"
> > > > >> > > REST
> > > > >> > > > > > >> > > interface helps user community. This also helps
> us
> > to
> > > > >> > > integrate
> > > > >> > > > > well
> > > > >> > > > > > >> > > with external management and provisioning tools.
> > > > >>Apache
> > > > >> > Kafka
> > > > >> > > > > > >>release
> > > > >> > > > > > >> > > with Java clients + REST interface is sufficient
> > for
> > > > >>most
> > > > >> of
> > > > >> > > the
> > > > >> > > > > > >>user
> > > > >> > > > > > >> > > deployments/requirements. This helps users to
> deal
> > > with
> > > > >> less
> > > > >> > > > > number
> > > > >> > > > > > >> > > of distributions/builds.
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > Thanks,
> > > > >> > > > > > >> > > Manikumar
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps <
> > > > >> j...@confluent.io
> > > > >> > >
> > > > >> > > > > wrote:
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > > Hey guys,
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > There's already a REST interface maintained as
> a
> > > > >> separate
> > > > >> > > > > > >> project--it's
> > > > >> > > > > > >> > > > open source and apache licensed and actively
> > > > >>maintained
> > > > >> (
> > > > >> > > > > > >> > > > https://github.com/confluentinc/kafka-rest).
> > What
> > > is
> > > > >> > wrong
> > > > >> > > > with
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > GitHub - confluentinc/kafka-rest: REST Proxy for Kafka
> > > > >> > > > > > github.com
> > > > >> > > > > > The Kafka REST Proxy provides a RESTful interface to a
> > Kafka
> > > > >> > cluster.
> > > > >> > > > It
> > > > >> > > > > > makes it easy to produce and consume messages, view the
> > > state
> > > > >>of
> > > > >> > the
> > > > >> > > > > > cluster, and ...
> > > > >> > > > > >
> > > > >> > > > > > >> that?
> > > > >> > > > > > >> > > You
> > > > >> > > > > > >> > > > mentioned that there was some compatibility
> > > concern,
> > > > >but
> > > > >> > > > > > >> compatibility
> > > > >> > > > > > >> > > has
> > > > >> > > > > > >> > > > to do with the consumer protocol guarantees not
> > the
> > > > >repo
> > > > >> > the
> > > > >> > > > > code
> > > > >> > > > > > >>is
> > > > >> > > > > > >> > in,
> > > > >> > > > > > >> > > > right? Not sure that concern makes sense.
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > We could argue for adding pretty much anything
> > and
> > > > >> > > everything
> > > > >> > > > in
> > > > >> > > > > > >>the
> > > > >> > > > > > >> > > > ecosystem page in Kafka itself but I'm not sure
> > > that
> > > > >> would
> > > > >> > > > make
> > > > >> > > > > > >>the
> > > > >> > > > > > >> > > project
> > > > >> > > > > > >> > > > more agile.
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > -Jay
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> > > > >> > > > > > >> manikumar.re...@gmail.com
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > > > wrote:
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > > > > Hi Kafka Devs,
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > I created KIP-80 to add Kafka REST Server to
> > > Kafka
> > > > >> > > > Repository.
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > There are already open-source alternatives
> are
> > > > >> > available.
> > > > >> > > > But
> > > > >> > > > > > >>we
> > > > >> > > > > > >> > would
> > > > >> > > > > > >> > > > > like to add REST server that
> > > > >> > > > > > >> > > > > many users ask for under Apache Kafka repo.
> > Many
> > > > >>data
> > > > >> > > Infra
> > > > >> > > > > > >>tools
> > > > >> > > > > > >> > comes
> > > > >> > > > > > >> > > > up
> > > > >> > > > > > >> > > > > with Rest Interface.
> > > > >> > > > > > >> > > > > It is useful to have inbuilt Rest API support
> > for
> > > > >> > Produce,
> > > > >> > > > > > >>Consume
> > > > >> > > > > > >> > > > messages
> > > > >> > > > > > >> > > > > and admin interface for
> > > > >> > > > > > >> > > > > integrating with external management and
> > > > >>provisioning
> > > > >> > > > > tools.This
> > > > >> > > > > > >> will
> > > > >> > > > > > >> > > > also
> > > > >> > > > > > >> > > > > allow the maintenance of
> > > > >> > > > > > >> > > > > REST server and adding new features makes it
> > easy
> > > > >> > because
> > > > >> > > > > apache
> > > > >> > > > > > >> > > > community.
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > The KIP wiki is the following:
> > > > >> > > > > > >> > > > > https://cwiki.apache.org/
> > > > >> confluence/display/KAFKA/KIP-
> > > > >> > > > > > >> > > > > 80%3A+Kafka+Rest+Server
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > Your comments and feedback are welcome.
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > > > Thanks,
> > > > >> > > > > > >> > > > > Manikumar
> > > > >> > > > > > >> > > > >
> > > > >> > > > > > >> > > >
> > > > >> > > > > > >> > >
> > > > >> > > > > > >> > --
> > > > >> > > > > > >> > Thanks,
> > > > >> > > > > > >> > Neha
> > > > >> > > > > > >> >
> > > > >> > > > > > >>
> > > > >> > > > > > The information contained in this email is strictly
> > > > >>confidential
> > > > >> > and
> > > > >> > > > for
> > > > >> > > > > > the use of the addressee only, unless otherwise
> indicated.
> > > If
> > > > >you
> > > > >> > are
> > > > >> > > > not
> > > > >> > > > > > the intended recipient, please do not read, copy, use or
> > > > >disclose
> > > > >> > to
> > > > >> > > > > others
> > > > >> > > > > > this message or any attachment. Please also notify the
> > > sender
> > > > >>by
> > > > >> > > > replying
> > > > >> > > > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > > > >delete
> > > > >> > the
> > > > >> > > > > email
> > > > >> > > > > > and any copies of it. Opinions, conclusion (etc) that do
> > not
> > > > >> relate
> > > > >> > > to
> > > > >> > > > > the
> > > > >> > > > > > official business of this company shall be understood as
> > > > >>neither
> > > > >> > > given
> > > > >> > > > > nor
> > > > >> > > > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > > (a
> > > > >> > company
> > > > >> > > > > > registered in England and Wales, company number
> 04008957)
> > > and
> > > > >>IG
> > > > >> > > Index
> > > > >> > > > > > Limited (a company registered in England and Wales,
> > company
> > > > >> number
> > > > >> > > > > > 01190902). Registered address at Cannon Bridge House, 25
> > > > >>Dowgate
> > > > >> > > Hill,
> > > > >> > > > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > > > >195355)
> > > > >> > and
> > > > >> > > > IG
> > > > >> > > > > > Index Limited (register number 114059) are authorised
> and
> > > > >> regulated
> > > > >> > > by
> > > > >> > > > > the
> > > > >> > > > > > Financial Conduct Authority.
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > > The information contained in this email is strictly
> confidential
> > > and
> > > > >> for
> > > > >> > > the use of the addressee only, unless otherwise indicated. If
> > you
> > > > >>are
> > > > >> not
> > > > >> > > the intended recipient, please do not read, copy, use or
> > disclose
> > > to
> > > > >> > others
> > > > >> > > this message or any attachment. Please also notify the sender
> by
> > > > >> replying
> > > > >> > > to this email or by telephone (+44(020 7896 0011) and then
> > delete
> > > > >>the
> > > > >> > email
> > > > >> > > and any copies of it. Opinions, conclusion (etc) that do not
> > > relate
> > > > >>to
> > > > >> > the
> > > > >> > > official business of this company shall be understood as
> neither
> > > > >>given
> > > > >> > nor
> > > > >> > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > > >>company
> > > > >> > > registered in England and Wales, company number 04008957) and
> IG
> > > > >>Index
> > > > >> > > Limited (a company registered in England and Wales, company
> > number
> > > > >> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > > > >>Hill,
> > > > >> > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > > > >>and
> > > > >> IG
> > > > >> > > Index Limited (register number 114059) are authorised and
> > > regulated
> > > > >>by
> > > > >> > the
> > > > >> > > Financial Conduct Authority.
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Nacho (Ignacio) Solis
> > > > >> Kafka
> > > > >> nso...@linkedin.com
> > > > >>
> > > >
> > > >
> > > >
> > >
> >
> > --
> >
> >
> > This email, including attachments, is private and confidential. If you
> have
> > received this email in error please notify the sender and delete it from
> > your system. Emails are not secure and may contain viruses. No liability
> > can be accepted for viruses that might be transferred by this email or
> any
> > attachment. Any unauthorised copying of this message or unauthorised
> > distribution and publication of the information contained herein are
> > prohibited.
> >
> > 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
> > Registered in England and Wales. Registered No. 04843573.
> >
>
--
This email, including attachments, is private and confidential. If you have
received this email in error please notify the sender and delete it from
your system. Emails are not secure and may contain viruses. No liability
can be accepted for viruses that might be transferred by this email or any
attachment. Any unauthorised copying of this message or unauthorised
distribution and publication of the information contained herein are
prohibited.
7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.
