Thanks for the feedback. > Why not take more time to develop the project and get the project into a more mature stage first? There is no harm to develop the project in that way.
Yes, that's what I'm planning to do. The plan is to move the project to https://github.com/datastax/reactive-pulsar and open it up for contributors which have signed a CLA that allows moving the project to Apache in the future. Does that make sense? For anyone looking for information about the project, the recording of my talk which describe it, is now available on YouTube, it's https://www.youtube.com/watch?v=scJkuSt280o . The slides are available at https://www.slideshare.net/Pivotal/reactive-applications-with-apache-pulsar-and-spring-boot . -Lari On Wed, Sep 8, 2021 at 12:37 PM Sijie Guo <guosi...@gmail.com> wrote: > It seems that there are already interests in contributing to the project. > Why not take more time to develop the project and get the project into a > more mature stage first? There is no harm to develop the project in that > way. > > - Sijie > > On Thu, Sep 2, 2021 at 12:06 AM Lari Hotari <l...@hotari.net> wrote: > > > On Thu, Sep 2, 2021 at 9:51 AM Enrico Olivelli <eolive...@gmail.com> > > wrote: > > > > > As Sijie said it is probably too early to bring it in, as the project > is > > > still very new. > > > On the other hand, as far as it is a single person project, I am not > sure > > > how many people will want to use it in production > > > and we want to bring Pulsar to a wider audience. > > > > > > > Reactive Streams is the future of streaming applications on the JVM. > > It won't be a single person project. There's already Pulsar community > > members that are looking forward to contributing. > > Here's one example: > > https://github.com/apache/pulsar/issues/10437#issuecomment-869928272 > > I have also another person that has contacted me and would like to start > > contributing. > > > > I have understood that it is a different process to accept an existing > > project with multiple contributors to Apache. > > Sijie and Enrico, please elaborate more about the concerns you have about > > the project being new and that I'm the author of it. > > I'd like to understand what problem it causes that the project is new. > > Isn't that a benefit? > > Let's discuss more about your concerns to gain more understanding. > > > > -Lari > > > > > > > > > > > > > > some answers to other comments inline below > > > > > > Enrico > > > > > > Il giorno gio 2 set 2021 alle ore 08:26 Rajan Dhabalia < > > > rdhaba...@apache.org> > > > ha scritto: > > > > > > > *My two cents about developing and contributing adapters/connectors > for > > > > Apache Pulsar. Apache Pulsar was open-sourced with Spark and Storm > > > adapters > > > > initially and then such adapter families evolved with multiple other > > > > adapters, connectors and IO connectors (sink/source) into apache > pulsar > > > git > > > > repo. Such adapters help pulsar for its adaption and user growth. > > > However, > > > > after adding more adapters in the pulsar repo, we realized two major > > > > issues: One of the issues, has made Pulsar git repo bulky which > > increases > > > > build time and sometimes it’s difficult to build and implement > > > integration > > > > tests. Another issue is the maintenance of such adapters with new > > > releases > > > > of the dependent systems and bug fixes for multiple versions. > > Therefore, > > > we > > > > eventually created a separate repository to manage adapters > separately > > > into > > > > pulsar-adapters gitrepo. So, one of the takeaways from past > experience, > > > if > > > > we absolutely need an adapter then we can use a pulsar-adapter > gitrepo > > > and > > > > we should evaluate the feasibility of maintaining such adapters with > > > their > > > > different versions and their interest in the community. Thanks,Rajan* > > > > > > > > On Wed, Sep 1, 2021 at 10:59 PM Lari Hotari <l...@hotari.net> wrote: > > > > > > > > > Replies inline below: > > > > > > > > > > On Thu, Sep 2, 2021 at 3:37 AM Sijie Guo <guosi...@gmail.com> > wrote: > > > > > > > > > > > Lari, > > > > > > > > > > > > Thank you for bringing this up! > > > > > > > > > > > > In general, I would love to see this being accepted to the > project. > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > > > However, in the past, there were other language clients > contributed > > > to > > > > > the > > > > > > project. They were not accepted because the PMC doesn't have > enough > > > > > > bandwidth to staff it. > > > > > > > > > > I think that the community is still growing and also the PMC, if we > have > > > enough people that are willing to support this > > > then I feel we can accept it. > > > In particular Lari is already part of the Community as committer and he > > > will take care of the project. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/pulsar/wiki/PIP-77%3A-Contribute-Supernova-to-Apache-Pulsar > > > > > > is one of the examples. > > > > > > > > > > > > > > > > I don't think that Reactive Streams is comparable to this Supernova > > > > > example. That is an example of a very marginal language. > > > > > > > > > > Reactive Streams support is a mainstream technology that originates > > > from > > > > > Netflix. It's not about a single programming language. > > > > > The Reactive Streams is an interoperability specification and there > > are > > > > > multiple implementations for the JVM. > > > > > > > > > > For example, this Reaction Pulsar Adapter library uses Project > > Reactor > > > as > > > > > the implementation, but the library can be used together with Akka > > > > Streams, > > > > > RxJava 3, Vert.x, SmallRye Mutiny (RedHat/Quarkus) and others. > > > > > > > > > > This library is not about a single programming language, it's about > > all > > > > > languages and platforms on the JVM that support Reactive Streams. > > > > > It could be an extension directly to the Pulsar Java Client. I just > > > > thought > > > > > that maintaining the project would be easier when it's a separate > > > > > repository. > > > > > If that is causing the staffing issue that you referred to being > the > > > > > showstopper issue, I could also restructure the project so that it > > > could > > > > be > > > > > added as modules to apache/pulsar repository. > > > > > > > > > > There's a short summary in > > > > > > > > > > > > > > > > > > > > https://www.reactive.foundation/post/why-reactive-principles-ensure-system-scalability-with-josh-long > > > > > about the value proposition of Reactive Streams and why it matters. > > > > > The recent Netflix Techblog explicitly mentions Reactive Streams in > > the > > > > > blog post about Netflix's device management platform: > > > > > > > > > > > > > > > > > > > > https://netflixtechblog.com/towards-a-reliable-device-management-platform-4f86230ca623 > > > > > "Back-Pressure Support - Because the processing workload varies > > > > > significantly over time, the solution must first and foremost scale > > > with > > > > > the message load by providing back-pressure support as defined in > the > > > > > Reactive Streams specification — in other words, the solution > should > > be > > > > > able to switch between push and pull-based back-pressure models > > > depending > > > > > on the downstream being able to cope with the message production > rate > > > or > > > > > not." > > > > > > > > > > > > > > > > Also, the project is still relatively new and there is only one > > > > > > contributor. > > > > > > > > > > > > > > > > > > Everything has a beginning. :) > > > > > I was assuming that this would be optimal that there is only one > > > > > contributor and that other contributions could contaminate the IPR > > and > > > > make > > > > > it harder to move to the Apache Pulsar project later. > > > > > As mentioned above, this is more like an extension to the Pulsar > Java > > > > > Client and I could restructure the project so that it could be > added > > as > > > > > modules to the apache/pulsar repository if the separate repository > > > > > causes the barrier. > > > > > > > > > > > > > > > > Hence, my recommendation is to grow the contributors for this > > project > > > > so > > > > > > that there are enough contributors to work on that project. Once > > > there > > > > > are > > > > > > sufficient resources and contributors, the PMC can revisit this > > > > proposal > > > > > > and see if the PMC is able to staff it. > > > > > > > > > > > > > > > > Did the PMC already vote on this? :) I hope we could have the > > > discussion > > > > > before voting. > > > > > Adding support for Reactive Streams will add a lot of value to > Apache > > > > > Pulsar. > > > > > > > > > > I'm talking about the value of Reactive Streams for Apache Pulsar > > > > > application developers TODAY AT 11:05PT in > > > > > my SpringOne talk "Reactive Applications with Apache Pulsar and > > Spring > > > > > Boot" > > > > > > > > > > > > > > > > > > > > https://springone.io/2021/sessions/reactive-applications-with-apache-pulsar-and-spring-boot > > > > > It is free to attend, https://springone.io to register. There's > also > > > an > > > > > AMA > > > > > session immediately after the session Q&A. > > > > > Please join. Come and see! > > > > > > > > > > I'm looking forward to hearing more voices from the Apache Pulsar > > > > community > > > > > in this email discussion. > > > > > > > > > > BR, Lari > > > > > > > > > > > > > > > > > > > > > > - Sijie > > > > > > > > > > > > > > > > > > On Wed, Sep 1, 2021 at 4:31 AM Lari Hotari <lhot...@apache.org> > > > wrote: > > > > > > > > > > > > > Dear Apache Pulsar community members, > > > > > > > > > > > > > > I have developed a Reactive Streams adapter for the Apache > Pulsar > > > > Java > > > > > > > Client. It uses Project Reactor as the Reactive Streams > > > > implementation. > > > > > > > > > > > > > > The repository is here: > > https://github.com/lhotari/reactive-pulsar > > > > > > > and library is available in Maven Central: > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://search.maven.org/search?q=g:com.github.lhotari%20a:reactive-pulsar* > > > > > > > > > > > > > > I would like to contribute this library to the Apache Pulsar > > > project. > > > > > > > I would suggest that the library is committed to a separate > > > > repository, > > > > > > for > > > > > > > example https://github.com/apache/pulsar-reactive . > > > > > > > > > > > > > > # Motivation > > > > > > > > > > > > > > There's a need to "go reactive from end-to-end" when building > > > modern > > > > > > > reactive applications with platforms such as Spring Reactive. > > > > > > > > > > > > > > There are ways to adapt the Apache Pulsar Java client async API > > > calls > > > > > to > > > > > > > Reactive Streams with a few lines of code. However, a lot will > be > > > > > missing > > > > > > > and achieving the complete solution will require much more > > effort. > > > > > > > > > > > > > > The primary goal of this library is to provide a Reactive > Streams > > > > > adapter > > > > > > > that wraps the Apache Pulsar Java client with Reactive APIs > > > > > > > for sending, reading and consuming messages. > > > > > > > The secondary goal is to provide a Spring Boot starter for > easily > > > > > > adopting > > > > > > > the power of Apache Pulsar for Spring Boot applications. > > > > > > > > > > > > > > It is currently out of scope for this library to build a > reactive > > > > > client > > > > > > > all the way from the binary protocol handling. > > > > > > > This library builds upon the existing Apache Pulsar Java client > > and > > > > > uses > > > > > > > the async API for operations. > > > > > > > > > > > > > > # Features > > > > > > > > > > > > > > - Simple and intuitive reactive APIs for sending, reading and > > > > consuming > > > > > > > messages. > > > > > > > - Non-blocking backpressure support for the provided reactive > > APIs > > > > > > > - Pulsar client resource lifecycle management that integrates > > with > > > > the > > > > > > > Reactive programming model > > > > > > > - Additional producer caching that caches producers. > > > > > > > - Useful for API backends since the Pulsar client producer > > > resource > > > > > can > > > > > > > be shared and cached across multiple REST calls. > > > > > > > - Support for in-order parallel (concurrent) processing at the > > > > > > application > > > > > > > level > > > > > > > - I/O intensive message processing can leverage Project > > Reactor's > > > > > > > capabilities to efficiently > > > > > > > handle thousands of simultaneous I/O calls to external REST > > > APIs > > > > > and > > > > > > > resources that have Reactive clients/drivers. > > > > > > > - In-order parallel processing support can retain key-order > at > > > the > > > > > > level > > > > > > > of a single consumer. > > > > > > > > > > > > > > I'm presenting the library and it's features tomorrow on > > September > > > > 2nd > > > > > in > > > > > > > the SpringOne virtual conference, which is free to attend. > > > > > > > The session details and schedule are at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://springone.io/2021/sessions/reactive-applications-with-apache-pulsar-and-spring-boot > > > > > > > . > > > > > > > > > > > > > > # Licenses > > > > > > > > > > > > > > The Reactive Pulsar Adapter library is under the Apache Licence > > > > Version > > > > > > 2.0 > > > > > > > . The main dependencies are reactor-core > > > > > > > (https://github.com/reactor/reactor-core) and jctools-core ( > > > > > > > https://github.com/JCTools/JCTools) which are also > > > > > > > ASL 2.0 licensed. > > > > > > > There's also a dependency to Caffeine ( > > > > > > > https://github.com/ben-manes/caffeine) > > > > > > > and the Spring Boot Starter > > > > > > > depends on Spring Boot and Spring Framework. These are all ASL > > 2.0 > > > > > > licensed > > > > > > > libraries. > > > > > > > The additional Spring integration testing support library > depends > > > on > > > > > > > Testcontainers, which is also ASL 2.0 licensed. > > > > > > > > > > > > > > > > > > > > > I'm looking forward to the discussion about this. > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > Lari Hotari > > > > > > > > > > > > > > > > > > > > > > > > > > > >
