Re: [VOTE] Apache Pulsar Adapters Release 2.8.0 Candidate 1
Grade Jobs! Thanks Enrico. -- Thanks Xiaolong Ran Enrico Olivelli 于2021年6月26日周六 下午7:52写道： > This is the first release candidate for Apache Pulsar Adapters, version > 2.8.0. > > It fixes the following issues: > https://github.com/apache/pulsar-adapters/milestone/2?closed=1 > > *** Please download, test and vote on this release. This vote will stay > open > for at least 72 hours *** > > Note that we are voting upon the source (tag), binaries are provided for > convenience and are released only to Maven Central. > > Source and binary files: > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-adapters-2.8.0-candidate-1/ > > SHA-512 checksums: > > > 6f1f34fc454fdaa9b5065d0dce96d468d8b5081cff333564cb88dca7e9c9aba828bb7469bb8c3aa4d23fe653d1a03285418d9ba36d5617a39a783e0433d01dba > apache-pulsar-adapters-2.8.0-src.tar.gz > > Maven staging repo: > https://repository.apache.org/content/repositories/orgapachepulsar-1091/ > > The tag to be voted upon: > v2.8.0-candidate-1 (581fee3b68caaf74122e7051e9741494b57fcbd7) > https://github.com/apache/pulsar-adapters/releases/tag/v2.8.0-candidate-1 > > Pulsar's KEYS file containing PGP keys we use to sign the release: > https://dist.apache.org/repos/dist/dev/pulsar/KEYS > > Please download the source package, and follow the README to build the > Pulsar Adapters code > > Enrico Olivelli >
Re: [Doc] generate docs from code automatically
Thanks yu, it seems to be a good start, so that we can better ensure the consistency of the code and documentation, rather than relying on artificial methods to fix the consistency problem. Of course, I would be happy to participate if time permits. -- Thanks Xiaolong Ran Anonymitaet _ 于2021年7月7日周三 上午12:01写道： > Hi Pulsarers, > > > > With the rapid adoption of Pulsar, our community is providing many > features and enhancements for *Pulsar command-line tools* and *Pulsar > configuration*, that is great! > > > > Previously, the docs of them are updated *manually*, which is > time-consuming and error-prone. To improve the productivity, effectiveness, > and velocity of developing these contents, we are developing tools to > *generate docs from source code files automatically*. > > > > In this way, we can get the code and the doc to good enough quality on > both sides in parallel and then merge both together so they end up shipping > in the same release. > > > > Below is the comparison of before and after. Here takes the docs of > pulsar-client and pulsar-perf as examples. > > > > # pulsar-client > > - Previous: > https://pulsar.apache.org/docs/en/next/reference-cli-tools/#pulsar-client > > - Now: http://pulsar.apache.org/tools/pulsar-client/2.9.0-SNAPSHOT/ > > > > # pulsar-perf > > - Previous: > https://pulsar.apache.org/docs/en/next/reference-cli-tools/#pulsar-perf > > - Now: http://pulsar.apache.org/tools/pulsar-perf/2.9.0-SNAPSHOT/ > > > > Obviously, this is a huge project. So far, we have completed some tasks > and documented the progress and all details here: > https://docs.google.com/spreadsheets/d/1P6tePGmU1lEhGKY_hE3JWgJ1rUTBCmW7AshDICJZr3o/edit#gid=1030826289 > > > > Feel free to comment or CONTRIBUTE TOGETHER, we'd love your feedback here! > > > > P.S. Special thanks goes to tuteng, urfreespace, and 315157973 for making > this happen! > > > > Anonymitaet > > > > >
Re: [RESULT] [VOTE] Apache Pulsar Adapters Release 2.8.0 Candidate 1
Enrico - Thank you for working on that! - Sijie On Wed, Jul 7, 2021 at 1:49 AM Enrico Olivelli wrote: > > The vote passes with 3 binding votes. > > +1 Sijie, Rajan, Peng Hui > > I will promote the artifacts to Maven central. > I will ask help to PengHui in order to move the sources to the releases > repository > > I will start a task to modify the website and find a place to put a > reference to the download link and the release notes. > > Thank you to everyone who contributed to this first release ! > > Enrico > > Il giorno mar 6 lug 2021 alle ore 20:59 Sijie Guo ha > scritto: > > > Thanks! > > > > +1 for this release (despite there are issues reported by Penghui and > > Rajan) > > > > - checked the signatures > > - checked the staged artifacts > > > > - Sijie > > > > On Tue, Jul 6, 2021 at 8:42 AM Enrico Olivelli > > wrote: > > > > > > Sijie, > > > > > > Il giorno mar 6 lug 2021 alle ore 17:30 Sijie Guo > > ha > > > scritto: > > > > > > > Hi Enrico, > > > > > > > > Where is the release process documented? > > > > > > > > I can't find it in > > > > https://github.com/apache/pulsar/wiki/Release-process or in the > > > > pulsar-adapters wiki page. > > > > > > > > > > it is here, in the pulsar-adapters repo > > > https://github.com/apache/pulsar-adapters/wiki/Release-process > > > > > > I have created it by copying it from the Pulsar process. > > > The page is still work in progress, as I am adapting it while moving > > > forward the steps > > > > > > > > > Enrico > > > > > > > > > > > > > > - Sijie > > > > > > > > On Mon, Jul 5, 2021 at 2:34 AM Enrico Olivelli > > > > wrote: > > > > > > > > > > Any other vote please ? > > > > > > > > > > We need at least one more +1 from a PMC member > > > > > > > > > > Thanks > > > > > Enrico > > > > > > > > > > Il giorno ven 2 lug 2021 alle ore 01:24 Rajan Dhabalia < > > > > rdhaba...@apache.org> > > > > > ha scritto: > > > > > > > > > > > Sure, let's fix it as a part of the 2.9.0 release to build an > > adapter > > > > repo > > > > > > without building pulsar locally. > > > > > > > > > > > > +1 for me. > > > > > > - Validated signature and src code build. > > > > > > > > > > > > Thanks, > > > > > > Rajan > > > > > > > > > > > > On Thu, Jul 1, 2021 at 1:43 PM Enrico Olivelli < > > eolive...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Rajan, > > > > > > > Thanks for testing probably the instructions are not clear, see > > my > > > > > > answers > > > > > > > below > > > > > > > > > > > > > > Il Gio 1 Lug 2021, 21:27 Rajan Dhabalia > > ha > > > > > > scritto: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > Right now, Build is failing with the below error which > > prevents us > > > > from > > > > > > > > building the project with all test execution. README page also > > > > suggests > > > > > > > to > > > > > > > > build project by skipping tests (using -DskipTests options) > > > > > > > > > > > > > > > > > > > > > > The skip tests flag is to build Pulsar from > > > > > > > https://github.com/apache/pulsar > > > > > > > at tag v2.8.0 > > > > > > > Not to build the pulsar-adapters repo. > > > > > > > > > > > > > > This is because the integration tests jar is not deployed to > > Maven > > > > > > Central > > > > > > > as you can see in the error you posted. > > > > > > > > > > > > > > > > > > > > > > [ERROR] Failed to execute goal on project > > > > > > > > pulsar-kafka-compat-client-test: Could not resolve > > dependencies for > > > > > > > > project > > > > > > > > > org.apache.pulsar.tests:pulsar-kafka-compat-client-test:jar:2.8.0: > > > > > > > > Failure to find > > org.apache.pulsar.tests:integration:jar:tests:2.8.0 > > > > > > > > > > > > > > > > I think we should fix the build failure or disable the test > > module > > > > if > > > > > > > it's > > > > > > > > not correctly migrated from the Pulsar git-repository, rather > > than > > > > > > > > releasing and validating release with -DskipTests option. > > > > > > > > > > > > > > > > > > > > > > The tests are all passing and this is confirmed by the fact that > > the > > > > > > latest > > > > > > > PRs were validated in CI. > > > > > > > > > > > > > > During the past month we enabled CI and we fixed all the tests > > > > > > > > > > > > > > Can you try to build locally Pulsar at tag v2.8.0? > > > > > > > This way your maven local repository will contain the jar that is > > > > missing > > > > > > > > > > > > > > It is something we can fix for the the Pulsar release, but for > > 2.8.0 > > > > > > there > > > > > > > is no other way. > > > > > > > > > > > > > > Enrico > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Rajan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 29, 2021 at 3:27 AM PengHui Li > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Enrico, > > > > > > > > > sorry for the late reply > > > > > > > > > > > > > > > > > > 1. Checked the signature > > > > > > > > > 2. Build via the source code > > > >
Re: Re: Re: Re: Discussion about https://github.com/apache/pulsar/pull/11112
- Regarding the interface design:
One thing we need to confirm is whether we **don't want old runtime to execute
new function without proper initialization** or not.
If this rule must be followed, then one possible way is we define a separate
`HookFunction` interface which includes `setup`, `process`, `tearDown` methods,
and add support in the new runtime to handle this interface. Just as currently
the runtime handles `pulsar.Function` and `java.util.Function` separately. The
old runtime will not be able to recognize the new interface and thus won't
execute functions which need initialization.
Documentation and examples can be added to help user learn and understand this
new interface.
- Regarding the exception:
I think user is responsible for properly handle any checked exceptions and thus
the interface should not allow throwing any exception. For any unchecked
exceptions, the runtime/process will fail fast.
- Regarding the contract:
Given that checked exceptions are handled by user properly as described above,
the `setup` method must be successfully executed or exited. So, there's no need
to call the `tearDown` method.
On 2021/07/06 21:02:17 Enrico Olivelli wrote:
> Il Mar 6 Lug 2021, 21:30 Neng Lu ha scritto:
>
> > IMHO, The old runtime should not be able to run the new functions.
>
>
> This is not possible to enforce this because we cannot change old code.
>
> >
> >
> > New functions require resource initialization via hooks. If the actual
> > `setup()` method is not called (or only a default no-op one is called),
> > then the function is not properly initialized and there'll be problems if
> > they are executed.
> >
>
> I believe that using default methods is better, as it is clearer for new
> users.
>
> If you add a new interface then it is difficult for a new user to discover
> this feature.
>
> Additionally we have to define the contract for calling the teardown
> function.
> Will it be called in case of failure of the setup function?
>
> Are those functions allowed to throw exceptions. I believe it is good to
> let them throw Exception in order to not force the user to catch everything
> and re throw as a RuntimeException
>
> Enrico
>
>
> >
> > On 2021/07/06 19:03:09 Sijie Guo wrote:
> > > So there are two compatibility issues we need to consider here.
> > >
> > > 1) Old runtime to run new functions.
> > > 2) New runtime to run old functions.
> > >
> > > Making the methods with default no-op implementation will resolve 1).
> > > Is that correct?
> > >
> > > We can use reflection to check if the methods exist or not to solve 2),
> > no?
> > >
> > > - Sijie
> > >
> > > On Tue, Jul 6, 2021 at 11:34 AM Neng Lu wrote:
> > > >
> > > > I think the reason is for keeping the original `Function` unchanged,
> > so that existing implemented functions are not affected.
> > > >
> > > >
> > > > On 2021/07/06 03:34:49 Sijie Guo wrote:
> > > > > Thank you for starting the discussion!
> > > > >
> > > > > I have added this proposal to PIP-86:
> > > > >
> > https://github.com/apache/pulsar/wiki/PIP-86:-Pulsar-Functions:-Preload-and-release-external-resources
> > > > >
> > > > > I have one question: why do you introduce HookFunction? Why not just
> > add
> > > > > two default methods to the existing Functions API?
> > > > >
> > > > > - Sijie
> > > > >
> > > > > On Mon, Jul 5, 2021 at 6:49 PM 陈磊
> > wrote:
> > > > >
> > > > > > Motivation
> > > > > >
> > > > > > It is very useful in many scenarios to provide safe and convenient
> > > > > > capabilities for function's external resource initialization and
> > release.
> > > > > > In addition to the normal data processing path, it enables
> > functions to use
> > > > > > HookFunction to manage external resources
> > > > > >
> > > > > > At present, in order to process data, only the logic of resource
> > > > > > initialization -> processing -> release and shutdown can be
> > written in the
> > > > > > process() of Function. This method is complicated, insecure, and
> > > > > > unnecessary.
> > > > > >
> > > > > > Instead, we should have a new standard way for users to use
> > Function
> > > > > > easily and safely. Summarized as follows:
> > > > > >
> > > > > > Before Function starts, some resources only need to be initialized
> > once,
> > > > > > and there is no need to make various judgments in the process()
> > method of
> > > > > > the Function interface
> > > > > >
> > > > > > After closing the Function, in the process of using process(), you
> > need to
> > > > > > manually close the referenced external resources, which need to be
> > released
> > > > > > separately in the close() of javaInstance
> > > > > > API and Implementation Changes
> > > > > >
> > > > > > The organization of the function implementation hierarchy has been
> > added,
> > > > > > it currently looks like the following figure:
> > > > > >
> > > > > > Use Cases:
> > > > > >
> > > > > > Before transformation
> > > > > > public class DemoFunction implements Function{
[GitHub] [pulsar-dotpulsar] DrinkStar closed issue #79: How to set receiveQueueSize of consumer
DrinkStar closed issue #79: URL: https://github.com/apache/pulsar-dotpulsar/issues/79 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-dotpulsar] DrinkStar opened a new issue #79: How to set receiveQueueSize of consumer
DrinkStar opened a new issue #79: URL: https://github.com/apache/pulsar-dotpulsar/issues/79 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[RESULT] [VOTE] Apache Pulsar Adapters Release 2.8.0 Candidate 1
The vote passes with 3 binding votes. +1 Sijie, Rajan, Peng Hui I will promote the artifacts to Maven central. I will ask help to PengHui in order to move the sources to the releases repository I will start a task to modify the website and find a place to put a reference to the download link and the release notes. Thank you to everyone who contributed to this first release ! Enrico Il giorno mar 6 lug 2021 alle ore 20:59 Sijie Guo ha scritto: > Thanks! > > +1 for this release (despite there are issues reported by Penghui and > Rajan) > > - checked the signatures > - checked the staged artifacts > > - Sijie > > On Tue, Jul 6, 2021 at 8:42 AM Enrico Olivelli > wrote: > > > > Sijie, > > > > Il giorno mar 6 lug 2021 alle ore 17:30 Sijie Guo > ha > > scritto: > > > > > Hi Enrico, > > > > > > Where is the release process documented? > > > > > > I can't find it in > > > https://github.com/apache/pulsar/wiki/Release-process or in the > > > pulsar-adapters wiki page. > > > > > > > it is here, in the pulsar-adapters repo > > https://github.com/apache/pulsar-adapters/wiki/Release-process > > > > I have created it by copying it from the Pulsar process. > > The page is still work in progress, as I am adapting it while moving > > forward the steps > > > > > > Enrico > > > > > > > > > > - Sijie > > > > > > On Mon, Jul 5, 2021 at 2:34 AM Enrico Olivelli > > > wrote: > > > > > > > > Any other vote please ? > > > > > > > > We need at least one more +1 from a PMC member > > > > > > > > Thanks > > > > Enrico > > > > > > > > Il giorno ven 2 lug 2021 alle ore 01:24 Rajan Dhabalia < > > > rdhaba...@apache.org> > > > > ha scritto: > > > > > > > > > Sure, let's fix it as a part of the 2.9.0 release to build an > adapter > > > repo > > > > > without building pulsar locally. > > > > > > > > > > +1 for me. > > > > > - Validated signature and src code build. > > > > > > > > > > Thanks, > > > > > Rajan > > > > > > > > > > On Thu, Jul 1, 2021 at 1:43 PM Enrico Olivelli < > eolive...@gmail.com> > > > > > wrote: > > > > > > > > > > > Rajan, > > > > > > Thanks for testing probably the instructions are not clear, see > my > > > > > answers > > > > > > below > > > > > > > > > > > > Il Gio 1 Lug 2021, 21:27 Rajan Dhabalia > ha > > > > > scritto: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > Right now, Build is failing with the below error which > prevents us > > > from > > > > > > > building the project with all test execution. README page also > > > suggests > > > > > > to > > > > > > > build project by skipping tests (using -DskipTests options) > > > > > > > > > > > > > > > > > > > The skip tests flag is to build Pulsar from > > > > > > https://github.com/apache/pulsar > > > > > > at tag v2.8.0 > > > > > > Not to build the pulsar-adapters repo. > > > > > > > > > > > > This is because the integration tests jar is not deployed to > Maven > > > > > Central > > > > > > as you can see in the error you posted. > > > > > > > > > > > > > > > > > > > [ERROR] Failed to execute goal on project > > > > > > > pulsar-kafka-compat-client-test: Could not resolve > dependencies for > > > > > > > project > > > > > > > org.apache.pulsar.tests:pulsar-kafka-compat-client-test:jar:2.8.0: > > > > > > > Failure to find > org.apache.pulsar.tests:integration:jar:tests:2.8.0 > > > > > > > > > > > > > > I think we should fix the build failure or disable the test > module > > > if > > > > > > it's > > > > > > > not correctly migrated from the Pulsar git-repository, rather > than > > > > > > > releasing and validating release with -DskipTests option. > > > > > > > > > > > > > > > > > > > The tests are all passing and this is confirmed by the fact that > the > > > > > latest > > > > > > PRs were validated in CI. > > > > > > > > > > > > During the past month we enabled CI and we fixed all the tests > > > > > > > > > > > > Can you try to build locally Pulsar at tag v2.8.0? > > > > > > This way your maven local repository will contain the jar that is > > > missing > > > > > > > > > > > > It is something we can fix for the the Pulsar release, but for > 2.8.0 > > > > > there > > > > > > is no other way. > > > > > > > > > > > > Enrico > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > Rajan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 29, 2021 at 3:27 AM PengHui Li > > > > wrote: > > > > > > > > > > > > > > > Hi Enrico, > > > > > > > > sorry for the late reply > > > > > > > > > > > > > > > > 1. Checked the signature > > > > > > > > 2. Build via the source code > > > > > > > > 3. Passed all the tests locally > > > > > > > > > > > > > > > > The only one I found is when I unzip the package, it shows > me an > > > > > > > > apache-pulsar-2.8.0-src directory, > > > > > > > > Looks a little wired, I think it should be > > > > > > > > apache-pulsar-adaptors-2.8.0-src? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Penghui > > > > > > > > > > > > > > > > Enrico Olivelli 于2021年6月29日周二 > 下午1:55写道： > > > >
