Re: [DISCUSS] Allow the producer to connect to the topic with producer_request_hold backlog policy
> IMO, we should allow the producer to connect the topic. I actually think the current producer_request_hold feature works based on disconnecting a producer and only letting it connect when the topic no longer exceeds its quota. > It looks like we do not support the `producer_request_hold` semantics. We > just easily use the same behaviour like `producer_exception`. Maybe it's a > missing feature. I agree that the only references to the RetentionPolicy.producer_request_hold enum have to do with disallowing producer creation or with disconnecting the producer when the backlog is exceeded [0]. However, I think the feature does work if we look closer. The implementation is in the client (at least it is in the Java client). First, note that the only functional difference between producer_exception and producer_request_hold comes here [1] where two different exceptions are sent to the producer. Then, see that the producer handles these exceptions differently [2]. The producer fails the pending messages when the policy is producer_exception and the producer does nothing when the policy is producer_request_hold. In the second case, the producer will attempt to reconnect to the broker and will resend the messages that have been "held". It seems relevant to point out that the backlog quota's state is only changed on a 60 second interval by default (see backlogQuotaCheckIntervalInSeconds) and the default send timeout is 30 seconds. Therefore, many sends will likely timeout on the client side before the broker updates the topic's state to "writable" and lets the producer reconnect. To use this feature meaningfully, it might make sense to increase the send timeout. The primary advantage for this solution is that the broker does not need to hold a producer's messages in memory for some undefined time. I just checked, and we do not have this behavior documented in the pulsar binary protocol spec [3]. We should update the spec to indicate how this feature is supposed to work, assuming we keep it this way. Thanks, Michael [0] https://github.com/apache/pulsar/blob/d24d82780fd27a98c6cdbee28d756ee7d419495f/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BacklogQuotaManager.java#L104-L107 [1] https://github.com/apache/pulsar/blob/4c6989c4da6c0b18c9b0196630e03daf437cea68/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java#L1383-L1391 [2] https://github.com/apache/pulsar/blob/955dcd10ce28b996811e194c9ad852b06ab30aee/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java#L1735-L1753 [3] https://pulsar.apache.org/docs/next/developing-binary-protocol/ On Thu, Aug 11, 2022 at 6:52 AM Qiang Zhao wrote: > > Hi, Penghui > > I support your opinion. > > It looks like we do not support the `producer_request_hold` semantics. We > just easily use the same behaviour like `producer_exception`. Maybe it's a > missing feature. > > Best, > Mattison > > On 2022/08/11 05:28:25 PengHui Li wrote: > > Hi all, > > > > Pulsar has a backlog quota policy `producer_request_hold` which will hold > > the message > > publish request. It is very useful for some data sync scenarios. It looks > > like the producer > > waiting for the consumer to process the data. > > > > But the new producer is not allowed to connect to the topic after reaching > > the max backlog > > limitation with producer_request_hold policy. The producer will get > > `ProducerBlockedQuotaExceededError` error. This caused inconvenience to > > users, they need > > to have some retry to logic to try to create the producer again until the > > consumer acked more > > messages > > > > IMO, we should allow the producer to connect the topic. To keep the > > behavior consistent with > > the producer is already connected. > > > > WDYT? > > > > Best Regards, > > Penghui > >
[GitHub] [pulsar] michaeljmarshall added a comment to the discussion: Issue - Failed to enable ssl with JKS
GitHub user michaeljmarshall added a comment to the discussion: Issue - Failed to enable ssl with JKS Note that the `Failed to resolve network location` log only indicates that you have not configured racks for your bookies. It is not preventing TLS from working, and it is not attempting to do any DNS resolution, assuming you're using the broker.conf defaults. I think we'll likely need more of the logs to answer what is going wrong. Specifically, we'll need to see the full stack trace from the client, and we'll also need to see the full stack trace from the broker. Note that the final error is coming from this block: https://github.com/apache/pulsar/blob/a3f52891593e093c27b583094a1fbfd09bbbae1a/pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/http/AsyncHttpConnector.java#L263-L266. GitHub link: https://github.com/apache/pulsar/discussions/17065#discussioncomment-3380909 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] michaeljmarshall added a comment to the discussion: Tips - Workaround incorrect DNS resolutions on MacOS
GitHub user michaeljmarshall added a comment to the discussion: Tips - Workaround incorrect DNS resolutions on MacOS @tisonkun - thanks for your comment about DNS resolution. I've also found that an issue can come from how the `/etc/hosts` file is configured for your machine's hostname. I documented my findings here and solution: https://github.com/apache/pulsar/issues/15401#issuecomment-1177120312. GitHub link: https://github.com/apache/pulsar/discussions/17067#discussioncomment-3380770 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun edited a comment on the discussion: Issue - Failed to enable ssl with JKS
GitHub user tisonkun edited a comment on the discussion: Issue - Failed to enable ssl with JKS @eronrunner can you provide which version (os & pulsar) in used? OK. It seems Pulsar version=2.9.2. If it's on MacOS, DNS resolution can meet issue described in https://github.com/apache/pulsar/discussions/17067. GitHub link: https://github.com/apache/pulsar/discussions/17065#discussioncomment-3380701 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun added a comment to the discussion: Issue - Failed to enable ssl with JKS
GitHub user tisonkun added a comment to the discussion: Issue - Failed to enable ssl with JKS @eronrunner can you provide which version (os & pulsar) in used? GitHub link: https://github.com/apache/pulsar/discussions/17065#discussioncomment-3380701 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] nodece added a comment to the discussion: Issue - Failed to enable ssl with JKS
GitHub user nodece added a comment to the discussion: Issue - Failed to enable ssl with JKS It looks related to DNS. GitHub link: https://github.com/apache/pulsar/discussions/17065#discussioncomment-3380663 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
Re: [Discuss] PIP 198: Standardize PR Naming Convention using GitHub Actions
+1 on the customized one. Cheers momo-jun
Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic
> I suggest to include 'topic' in the flag, we have too many entities in > Pulsar Thanks, change it to `topicTwoPhaseDeletionEnabled`.
Re: [Discuss] PIP 198: Standardize PR Naming Convention using GitHub Actions
+1 on the customized one Thanks, Yunze > 2022年8月12日 00:25，Alexander Preuss > 写道： > > Hi together, > > Thank you for driving this topic! > I agree that our customized convention is better than the Angular one. > > +1 on the customized one > > Best, > Alex > > > > On Thu, Aug 11, 2022 at 11:02 AM Yu wrote: > >> Hi team, >> >> For PIP 198: Standardize PR Naming Convention using GitHub Actions [1], we've >> got many different suggestions on implementation details. Let's discuss >> them one by one. >> >> Discussion topic: >> >> For PR titles, which convention should we follow? >> - Angular convention [2] - Our existing convention (it's customized based >> on Angular) [3] The differences between Angular and ours >> are: 1. Definition 1.1 Property - Angular: [type] is required, [scope] is >> optional - Ours: [type] and [scope] are required 1.2 Content - Angular: ci, >> test, and docs belong to [type] - Ours: ci, test, and docs belong to >> [scope] because >> I think [type] defines "what action do you make" (eg. >> add/delete/update/...), >> while [scope] defines "where do you make changes". 2. Punctuation - >> Angular: parenthesis and exclaim points are used - Ours: brackets are used >> Comparison examples Taking existing Pulsar PR titles as >> examples: Example 1 - Angular: fix: Filter out already deleted entries >> again before sending messages to consumers - Ours: [fix][broker] Filter out >> already deleted entries again before sending messages to consumers Example >> 2 - Angular: ci: add flaky test issues and PRs to flaky test project - >> Ours: [feat][ci] Add flaky test issues and PRs to flaky test project >> Example 3 - Angular: docs: Document configuration added by PIP-145 doc - >> Ours: [improve][doc] Document configuration added by PIP-145 doc >> I prefer our customized convention because: - It makes PR >> titles more clear and self-explanatory. - No need to change users' habits >> since many people in the community have followed and gotten used to it for >> several months [4]. Feel free to comment. Thank you! >> [1] >> https://docs.google.com/document/d/1sJlUNAHnYAbvu9UtEgCrn_oVTnVc1M5nHC19x1bFab4/edit?pli=1 >> >> [2] >> https://github.com/angular/angular/blob/main/CONTRIBUTING.md#commit-message-header >> [3] >> https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.y8943h392zno >> [4] https://github.com/apache/pulsar/pulls >> >> Yu, Max, mangoGoForward >>
[GitHub] [pulsar] Anonymitaet added a comment to the discussion: Tips - Workaround incorrect DNS resolutions on MacOS
GitHub user Anonymitaet added a comment to the discussion: Tips - Workaround incorrect DNS resolutions on MacOS How about adding a `troubleshooting` or `QA` chapter to https://pulsar.apache.org/docs/next/getting-started-standalone at the end? The [`Install JDK on M1` chapter](https://pulsar.apache.org/docs/next/getting-started-standalone#install-jdk-on-m1) can also be in that chapter since we want to navigate users to the core steps of `getting Pulsar started` in the quickest way? Thoughts? GitHub link: https://github.com/apache/pulsar/discussions/17067#discussioncomment-3380435 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
Re: [Discuss] PIP 198: Standardize PR Naming Convention using GitHub Actions
Hi together, Thank you for driving this topic! I agree that our customized convention is better than the Angular one. +1 on the customized one Best, Alex On Thu, Aug 11, 2022 at 11:02 AM Yu wrote: > Hi team, > > For PIP 198: Standardize PR Naming Convention using GitHub Actions [1], we've > got many different suggestions on implementation details. Let's discuss > them one by one. > > Discussion topic: > > For PR titles, which convention should we follow? > - Angular convention [2] - Our existing convention (it's customized based > on Angular) [3] The differences between Angular and ours > are: 1. Definition 1.1 Property - Angular: [type] is required, [scope] is > optional - Ours: [type] and [scope] are required 1.2 Content - Angular: ci, > test, and docs belong to [type] - Ours: ci, test, and docs belong to > [scope] because > I think [type] defines "what action do you make" (eg. > add/delete/update/...), > while [scope] defines "where do you make changes". 2. Punctuation - > Angular: parenthesis and exclaim points are used - Ours: brackets are used > Comparison examples Taking existing Pulsar PR titles as > examples: Example 1 - Angular: fix: Filter out already deleted entries > again before sending messages to consumers - Ours: [fix][broker] Filter out > already deleted entries again before sending messages to consumers Example > 2 - Angular: ci: add flaky test issues and PRs to flaky test project - > Ours: [feat][ci] Add flaky test issues and PRs to flaky test project > Example 3 - Angular: docs: Document configuration added by PIP-145 doc - > Ours: [improve][doc] Document configuration added by PIP-145 doc > I prefer our customized convention because: - It makes PR > titles more clear and self-explanatory. - No need to change users' habits > since many people in the community have followed and gotten used to it for > several months [4]. Feel free to comment. Thank you! > [1] > https://docs.google.com/document/d/1sJlUNAHnYAbvu9UtEgCrn_oVTnVc1M5nHC19x1bFab4/edit?pli=1 > > [2] > https://github.com/angular/angular/blob/main/CONTRIBUTING.md#commit-message-header > [3] > https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.y8943h392zno > [4] https://github.com/apache/pulsar/pulls > > Yu, Max, mangoGoForward >
Re: [DISCUSS] Create a new Github Project to track the flaky tests
+1 It would be awesome to utilize the GitHub Project to track issues. BTW, seems this is a Classic Project[2], and the New Project[1] can provide more features than the Classic Project[2]. [1] https://github.com/apache/pulsar/projects?type=new [2] https://github.com/apache/pulsar/projects?type=classic Best, Max Xu On Thu, Aug 11, 2022 at 11:26 AM Zixuan Liu wrote: > +1 > > Thanks, > Zixuan > > Anon Hxy 于2022年8月11日周四 10:26写道： > > > +1 > > Good idea. We are on the way making CI more robust > > > > Thanks, > > Xiaoyu Hou > > > > PengHui Li 于2022年8月10日周三 16:35写道： > > > > > Hi all, > > > > > > For better tracking flaky test fix, I have tried to create a Github > > > Project under the Pulsar repo > > https://github.com/apache/pulsar/projects/11 > > > (It can be removed if we don't like this way) > > > This will help us to have a more intuitive view of all the flaky tests, > > > how many are in progress, in the review stage, and approved. > > > > > > The project is public for all the contributors, so if you want > > > to contribute some flaky tests fixes, > > > you can go to the Github Project to peek up items in the Todo column. > > > > > > And I also created a PR https://github.com/apache/pulsar/pull/17038 to > > > add the PRs and issues > > > with `flaky-tests` label to this project automatically. > > > > > > BTW, I also have some questions about the Github Project automation. As > > > the description of > > > column `Review in progress, it said the PR with request changes would > go > > > to this column > > > automatically. But it doesn't work. I'm not sure why. > > > [image: image.png] > > > > > > Best, > > > Penghui > > > > > > > > >
Questions about the release process
Hi all, Recently I'm working on the release of 2.8.4 and it's near the vote of the 1st candidate but I have some questions. From the tutorial [1] we can see, the 8th step is "Run the vote". However, the 7th step is "Write release notes", should we execute this step later? I see the 16th step is also "Write release notes" but the 16th step at the beginning of "Release workflow" section is "Update the site". In addition, I found the previous candidate [2] includes the docker images, which is not included in the template of the 8th step "Run the vote". It seems to be the 10th step "Publish Docker Images". It seems that the documents are not maintained well, which really makes me confused. Therefore, before voting for the 1st candidate, I want to get some clarifications from the mail list. [1] https://github.com/apache/pulsar/wiki/Release-process [2] https://lists.apache.org/thread/q0g5ko617rb77b1wqpxy94ks5mq48d88 Thanks, Yunze
[GitHub] [pulsar-helm-chart] danny-krueger opened a new issue, #285: Automatic certificate renewal does not work
danny-krueger opened a new issue, #285: URL: https://github.com/apache/pulsar-helm-chart/issues/285 **Describe the bug** The reissued certificates from the Cert Manager will not be used by Pulsar until the pods are rebooted. **To Reproduce** Steps to reproduce the behavior: 1. Activate Cert Manager with the internal issuer. 2. Wait until the certificate has expired. 3. The certificates have been updated, but the Java Services have not checked this and SSL errors occur in the Zookeeper. `ERROR org.apache.zookeeper.server.NettyServerCnxnFactory - Unsuccessful handshake with session 0x0` `2022-08-03T14:53:45,862+ [epollEventLoopGroup-7-2] WARN org.apache.zookeeper.server.NettyServerCnxnFactory - Exception caught io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[io.netty-netty-codec-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [io.netty-netty-transport-classes-epoll-4.1.74.Final.jar:4.1.74.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at java.lang.Thread.run(Thread.java:829) [?:?]` **Expected behavior** When the Cert Manager issues new certificates, Pulsar should also be aware of this and use the new certificates. **Additional context** The certificates have been correctly reissued by the Cert Manager. They were also correct in the Config Maps and in the Secrets. Also where we were in the pods per shell, the new correct SSL certificates were there. But since Pulsar itself was already running, it did not re-read them. We think that Java caches the certificates. **Quick fix** After all pods were restarted everything worked again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[COMMITTER ATTENTION] You can close stale issues as not planned
Hi, TL;DR: If you find an issue tagged as Stale or lifecycle/stale or generally inactive, be confident to close it as not planned. Close as not planned is a GitHub Issue feature[1] and here is an example[2]. The longer version: I started a discussion about the stale bot and stale issues/prs weeks ago[3]. Apart from suggestions in the thread, it's agreed that currently, committers omit Stale or lifecycle/stale labels when they triage an issue. After an offline discussion with @codelipenghui we agree that spending time on handling issues is better than writing more automation or rules. One thing to help us handle the backlog is close stale issues directly as not planned[1]. As committers can be confident close stale issues, we can significantly reduce the backlog. A good example is Apache SkyWalking who has less than 100 issues because they close stale issues frequently (total 4000+ issues vs. Pulsar total 5000+ issues, same order of magnitude). If we reach similar status, we don't have to worry about stale bot at all and even simply remove it - our committers should be able to handle such traffic. Don't worry if the issue is still valuable to continue or investigate. Since we ask issue creators to search before creating, anyone can bump a closed as not planned issue or reopen. If no one continues the thread, it's definitely valueless. Keep a large number of open issues while no one is actually interested in handling hurting the community instead of helping it. Close as not planned shows that the issue does get resolved, and that provides enough information to collaborate. Best, tison. [1] https://github.blog/changelog/2022-03-10-the-new-github-issues-march-10th-update/#%F0%9F%95%B5%F0%9F%8F%BD%E2%99%80%EF%B8%8F-issue-closed-reasons [2] https://github.com/apache/pulsar-site/issues/146 [3] https://lists.apache.org/thread/tv774jqohdpx8x0dymsskrd90xwwfvgp
[GitHub] [pulsar] tisonkun added a comment to the discussion: Tips - Workaround incorrect DNS resolutions on MacOS
GitHub user tisonkun added a comment to the discussion: Tips - Workaround incorrect DNS resolutions on MacOS BTW, this can be a troubleshooting doc for running Pulsar on MacOS. For the document part, cc @Anonymitaet @momo-jun Do we have a place to attach such notes? For the technical part, cc @lhotari @codelipenghui Is it possible we bundle these libs on condition? Although I can regard it as a platform-specific issue and should be resolved by related users. GitHub link: https://github.com/apache/pulsar/discussions/17067#discussioncomment-3374198 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] tisonkun created a discussion: Tips - Workaround incorrect DNS resolutions on MacOS
GitHub user tisonkun created a discussion: Tips - Workaround incorrect DNS resolutions on MacOS I'd like to share my experience of workaround incorrect DNS resolutions on MacOS for Pulsar standalone. When you run Pulsar standalone by `bin/pulsar standalone` on MacOS, you should always see the following warnings: ``` 2022-08-11T20:24:10,581+0800 [main] WARN io.netty.resolver.dns.DnsServerAddressStreamProviders - Can not find io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider in the classpath, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. ``` This is because we don't bundle the related netty resolver for DNS on MacOS. It may or may not cause an issue. I meet one issue today: ``` 2022-08-11T14:32:22,296+0800 [main] WARN io.netty.resolver.dns.DnsServerAddressStreamProviders - Can not find io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider in the classpath, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. 2022-08-11T14:32:24,381+0800 [main] ERROR org.apache.pulsar.functions.worker.PulsarWorkerService - Error Starting up in worker java.lang.ExceptionInInitializerError: null at io.netty.resolver.dns.UnixResolverDnsServerAddressStreamProvider.parseSilently(UnixResolverDnsServerAddressStreamProvider.java:80) ~[netty-resolver-dns-4.1.77.Final.jar:4.1.77.Final] ``` This error causes the standalone process to fail and exit. To workaround the issue, download the necessary libs and add them to the classpath. 1. Download [netty-resolver-dns-classes-macos-4.1.77.Final.jar](https://repo1.maven.org/maven2/io/netty/netty-resolver-dns-classes-macos/4.1.77.Final) and [netty-resolver-dns-native-macos-4.1.77.Final-osx-aarch_64.jar](https://repo1.maven.org/maven2/io/netty/netty-resolver-dns-native-macos/4.1.77.Final) (version and arch can be varied on your env). 2. Place these two libs under a chosen folder, said `/path/to/my/libs`. 3. Run Pulsar standalone with env var `PULSAR_EXTRA_CLASSPATH` which allows adding extra libs: ```sh PULSAR_EXTRA_CLASSPATH=/path/to/my/libs/* ./bin/pulsar standalone ``` The issue should be resolved atm :D GitHub link: https://github.com/apache/pulsar/discussions/17067 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] asafm added a comment to the discussion: Issue - Failed to enable ssl with JKS
GitHub user asafm added a comment to the discussion: Issue - Failed to enable ssl with JKS @hangc0276 ? GitHub link: https://github.com/apache/pulsar/discussions/17065#discussioncomment-3374082 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
Re: [DISCUSS] Allow the producer to connect to the topic with producer_request_hold backlog policy
Hi, Penghui I support your opinion. It looks like we do not support the `producer_request_hold` semantics. We just easily use the same behaviour like `producer_exception`. Maybe it's a missing feature. Best, Mattison On 2022/08/11 05:28:25 PengHui Li wrote: > Hi all, > > Pulsar has a backlog quota policy `producer_request_hold` which will hold > the message > publish request. It is very useful for some data sync scenarios. It looks > like the producer > waiting for the consumer to process the data. > > But the new producer is not allowed to connect to the topic after reaching > the max backlog > limitation with producer_request_hold policy. The producer will get > `ProducerBlockedQuotaExceededError` error. This caused inconvenience to > users, they need > to have some retry to logic to try to create the producer again until the > consumer acked more > messages > > IMO, we should allow the producer to connect the topic. To keep the > behavior consistent with > the producer is already connected. > > WDYT? > > Best Regards, > Penghui >
[GitHub] [pulsar] eronrunner edited a discussion: Issue - Failed to enable ssl with JKS
GitHub user eronrunner edited a discussion: Issue - Failed to enable ssl with
JKS
I install pulsar by helm chart and try to add the properties
brokerServicePortTls: "{{.Values.broker.ports.pulsarssl}}"
webServicePortTls: "{{.Values.broker.ports.https}}"
tlsEnabled: "true"
tlsCertificateFilePath: "/pulsar/certs/manager.crt"
tlsKeyFilePath: "/pulsar/certs/manager.key"
tlsTrustCertsFilePath: "/pulsar/certs/ca.crt"
tlsProtocols: "TLSv1.2,TLSv1.1"
tlsEnabledWithKeyStore: "true"
requireTrustedClientCertOnConnect: "true"
tlsKeyStoreType: JKS
tlsKeyStore: "{{.Values.tlsKeyStore.tlsKeyStore}}"
tlsKeyStorePassword: "{{.Values.tlsKeyStore.tlsKeyStorePassword}}"
tlsTrustStoreType: JKS
tlsTrustStore: "{{.Values.tlsKeyStore.tlsTrustStore}}"
tlsTrustStorePassword: "{{.Values.tlsKeyStore.tlsTrustStorePassword}}"
brokerClientTlsEnabled: "{{.Values.tlsKeyStore.brokerClientTlsEnabled}}"
brokerClientTlsEnabledWithKeyStore:
"{{.Values.tlsKeyStore.brokerClientTlsEnabledWithKeyStore}}"
brokerClientTlsTrustStoreType: JKS
brokerClientTlsTrustStore: "{{.Values.tlsKeyStore.brokerClientTlsTrustStore}}"
brokerClientTlsTrustStorePassword:
"{{.Values.tlsKeyStore.brokerClientTlsTrustStorePassword}}"
I also tried some ways by disabling the HTTP port and non-ssl service port,
but got the same things.
Please help!
Thank you
GitHub link: https://github.com/apache/pulsar/discussions/17065
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[Discuss] PIP 198: Standardize PR Naming Convention using GitHub Actions
Hi team, For PIP 198: Standardize PR Naming Convention using GitHub Actions [1], we've got many different suggestions on implementation details. Let's discuss them one by one. Discussion topic: For PR titles, which convention should we follow? - Angular convention [2] - Our existing convention (it's customized based on Angular) [3] The differences between Angular and ours are: 1. Definition 1.1 Property - Angular: [type] is required, [scope] is optional - Ours: [type] and [scope] are required 1.2 Content - Angular: ci, test, and docs belong to [type] - Ours: ci, test, and docs belong to [scope] because I think [type] defines "what action do you make" (eg. add/delete/update/...), while [scope] defines "where do you make changes". 2. Punctuation - Angular: parenthesis and exclaim points are used - Ours: brackets are used Comparison examples Taking existing Pulsar PR titles as examples: Example 1 - Angular: fix: Filter out already deleted entries again before sending messages to consumers - Ours: [fix][broker] Filter out already deleted entries again before sending messages to consumers Example 2 - Angular: ci: add flaky test issues and PRs to flaky test project - Ours: [feat][ci] Add flaky test issues and PRs to flaky test project Example 3 - Angular: docs: Document configuration added by PIP-145 doc - Ours: [improve][doc] Document configuration added by PIP-145 doc I prefer our customized convention because: - It makes PR titles more clear and self-explanatory. - No need to change users' habits since many people in the community have followed and gotten used to it for several months [4]. Feel free to comment. Thank you! [1] https://docs.google.com/document/d/1sJlUNAHnYAbvu9UtEgCrn_oVTnVc1M5nHC19x1bFab4/edit?pli=1 [2] https://github.com/angular/angular/blob/main/CONTRIBUTING.md#commit-message-header [3] https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit?pli=1#bookmark=id.y8943h392zno [4] https://github.com/apache/pulsar/pulls Yu, Max, mangoGoForward
[GitHub] [pulsar] eronrunner edited a discussion: Issue - Failed to enable ssl with JKS
GitHub user eronrunner edited a discussion: Issue - Failed to enable ssl with
JKS
I install pulsar by helm chart and try to add the properties
brokerServicePortTls: "{{.Values.broker.ports.pulsarssl}}"
webServicePortTls: "{{.Values.broker.ports.https}}"
tlsEnabled: "true"
tlsCertificateFilePath: "/pulsar/certs/manager.crt"
tlsKeyFilePath: "/pulsar/certs/manager.key"
tlsTrustCertsFilePath: "/pulsar/certs/ca.crt"
tlsProtocols: "TLSv1.2,TLSv1.1"
tlsEnabledWithKeyStore: "true"
requireTrustedClientCertOnConnect: "true"
tlsKeyStoreType: JKS
tlsKeyStore: "{{.Values.tlsKeyStore.tlsKeyStore}}"
tlsKeyStorePassword: "{{.Values.tlsKeyStore.tlsKeyStorePassword}}"
tlsTrustStoreType: JKS
tlsTrustStore: "{{.Values.tlsKeyStore.tlsTrustStore}}"
tlsTrustStorePassword: "{{.Values.tlsKeyStore.tlsTrustStorePassword}}"
brokerClientTlsEnabled: "{{.Values.tlsKeyStore.brokerClientTlsEnabled}}"
brokerClientTlsEnabledWithKeyStore:
"{{.Values.tlsKeyStore.brokerClientTlsEnabledWithKeyStore}}"
brokerClientTlsTrustStoreType: JKS
brokerClientTlsTrustStore: "{{.Values.tlsKeyStore.brokerClientTlsTrustStore}}"
brokerClientTlsTrustStorePassword:
"{{.Values.tlsKeyStore.brokerClientTlsTrustStorePassword}}"
I also tried some ways by disabling the HTTP port and non-ssl service port,
but got the same things.
Please help!
Thank you
GitHub link: https://github.com/apache/pulsar/discussions/17065
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] eronrunner created a discussion: Issue - Failed to enable ssl with JKS
GitHub user eronrunner created a discussion: Issue - Failed to enable ssl with
JKS
I install pulsar by helm chart and try to add the properties
brokerServicePortTls: "{{.Values.broker.ports.pulsarssl}}"
webServicePortTls: "{{.Values.broker.ports.https}}"
tlsEnabled: "true"
# TLS Settings
tlsCertificateFilePath: "/pulsar/certs/manager.crt"
tlsKeyFilePath: "/pulsar/certs/manager.key"
tlsTrustCertsFilePath: "/pulsar/certs/ca.crt"
# tlsAllowInsecureConnection: "false"
tlsProtocols: "TLSv1.2,TLSv1.1"
## Enable KeyStore type
tlsEnabledWithKeyStore: "true"
requireTrustedClientCertOnConnect: "true"
## key store
tlsKeyStoreType: JKS
tlsKeyStore: "{{.Values.tlsKeyStore.tlsKeyStore}}"
tlsKeyStorePassword: "{{.Values.tlsKeyStore.tlsKeyStorePassword}}"
## trust store
tlsTrustStoreType: JKS
tlsTrustStore: "{{.Values.tlsKeyStore.tlsTrustStore}}"
tlsTrustStorePassword: "{{.Values.tlsKeyStore.tlsTrustStorePassword}}"
## internal client/admin-client config
brokerClientTlsEnabled: "{{.Values.tlsKeyStore.brokerClientTlsEnabled}}"
brokerClientTlsEnabledWithKeyStore:
"{{.Values.tlsKeyStore.brokerClientTlsEnabledWithKeyStore}}"
brokerClientTlsTrustStoreType: JKS
brokerClientTlsTrustStore: "{{.Values.tlsKeyStore.brokerClientTlsTrustStore}}"
brokerClientTlsTrustStorePassword:
"{{.Values.tlsKeyStore.brokerClientTlsTrustStorePassword}}"
I also tried some ways by disabling the HTTP port and non-ssl service port,
but got the same things.
Please help!
Thank you
GitHub link: https://github.com/apache/pulsar/discussions/17065
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[RESULT][VOTE] PIP 198: Standardize PR Naming Convention using GitHub Actions
Hi team, Thanks for all your suggestions! I'll move PIP 198 forward since we get 3 binding votes (Penghui, Lari, Michael) [1]. [1] https://lists.apache.org/thread/m7hfgpdjthrk8ogz3mnkprs018tkkc0m Yu
Re: [DISCUSS] PIP-186: Introduce two phase deletion protocol based on system topic
Il Sab 23 Lug 2022, 03:26 Yan Zhao ha scritto: > Yes, wo define `twoPhaseDeletionEnabled` in the Service Configuration. > Thanks. I suggest to include 'topic' in the flag, we have too many entities in Pulsar Enrico >
Re: [DISCUSS] PIP 197: Add Schema hash and equals to public API
A Schema instance is stateful (and contians mutable state) and in the end it contains a reference to the PulsarClient instance that is using it. Look at how AutoConsume schema works for instance. I don't think it is possible to support such API in the general case. If you want to use it as a key in a map then I suggest you to create a wrapper in your code. Enrico Il Ven 5 Ago 2022, 14:34 Alexander Preuss ha scritto: > Hello Pulsar community, > > I would like to start a discussion about PIP 197: Add Schema hash and > equals to public API. > You can find the proposal at https://github.com/apache/pulsar/issues/16959 > as well as pasted below. > > Looking forward to hearing your thoughts, > Alex > > ## Motivation > > Currently, the `Schema` interface in the public client-api does not provide > access to a sensible hash function. The fallback to Java’s object-equality > makes it unfit for use in most hash-based collections. For example, it > prevents usage as a key in a cache. > Further, the lack of a reliable equals function means that there is no way > to identify if two schemas are the same thing. > > ## Goal > > The goal of this proposal is to provide a sensible `hashCode` and `equals` > implementation for Schema as part of the public API. > > Currently, pulsar-common contains `SchemaHash`, a wrapper class that exists > to solve the aforementioned problems. However, `SchemaHash` is not part of > the public API, so users should not depend on it. > > > ## API Changes > > There is no further change required as moving `SchemaHash` from > pulsar-common to the public API. The only further change could be to > re-think the class name, as the wrapper offers more than just a schemas > hash. > > ## Implementation > > Move `SchemaHash` from pulsar-common > `org.apache.pulsar.common.protocol.schema` package into pulsar-client-api > `org.apache.pulsar.common.schema` package. > > > ## Reject Alternatives > > Providing default methods for `equals` and `hashCode` directly on the > `Schema` interface is not possible because Java prohibits overriding the > base Object methods. > Another option would be to provide the `hashCode` and `equals` > functionality through similarly-named default methods that could be used by > any `Schema` implementation. The drawback of this idea is that it requires > developers to override the equals and hashCode to use these provided > methods, as well as possibly polluting the interface. >
