Re: [VOTE] [PIP-204] Extensions for BrokerInterceptor
+1 (non-binding) Thanks, Xiaoyu Hou Aloys Zhang 于2022年8月30日周二 19:02写道： > Hi devs, > > This is the VOTE thread for PIP-204 Extensions for BrokerInterceptor. > > PIP issue: https://github.com/apache/pulsar/issues/17267 > > Discussion thread: > https://lists.apache.org/thread/3zgpbxffo7gzsb5mdh0sk2rq27bgw2bo > > This voting will open for at least 72h. > > > > Best, > Aloys >
Pulsar Community Meeting Notes 2022/09/01, (8:30 AM PST)
Hi Pulsar Community, Here are the meeting notes from today's community meeting. Thanks to all who participated! As you'll see, we had a very long discussion about the idea of replacing bundles. The other two topics were on broker metadata cache evictions and the pulsar protocol. Disclaimer: if something is misattributed or misrepresented, please send a correction to this list. Source google doc: https://docs.google.com/document/d/19dXkVXeU2q_nHmkG8zURjKnYlvD96TbKf5KjYyASsOE Thanks, Michael 2022/09/01, (8:30 AM PST) - Attendees: - Matteo Merli - Lari Hotari - Enrico Olivelli - Andrey Yegorov - Dave Fisher - Heesung Sohn - Ayman Khalil - Michael Marshall - Asaf Mesika - Christophe Bornet - Discussions/PIPs/PRs (Generally discussed in order they appear) - Lari: the mailing list discussion about namespace bundles and the long term plan. Matteo: I would like to remove them. They can be very annoying. They have to be big enough to be working, and small enough to not cause imbalance problems. There is an order of magnitude difference in the state by only managing them as groups, not as individuals. If you have millions of topics, how can a single broker keep track of everything? Dave: could part of the problem be that part of the problem is that adding the broker and growing topics, you encounter bundle splits, is the problem splitting them? Matteo: the split is a way to maintain them. If you have a bundle with too many topics, it is too large for even distribution of load. Dave: what if you could allocate a bunch of bundles at once? Michael: you can do that. Matteo: the problem there is that if you don’t need that, then it’s a waste of resources. It’s not terrible, but worse case is when you have one topic per bundle. If you know your use case, you can create the namespace with the “right” number of bundles. The system is ideally designed so that operators don’t need to configure these low level details. Dave: is that the point of the different bundle split algorithms, then? Matteo: yes. Lari: do we want to change the design at any point in the future? One case is topic level anti-affinity rules for a very high volume partitioned topic where you want to ensure each partition is on different brokers. Matteo: it is kind of possible today. If you want to maximize throughput, that should be the responsibility of the load manager. It is possible to have bundle level failure domains where bundles are split. Lari: since it is about bundles, it is extra complexity to ensure the topics are distributed because they could be in the same bundle depending on their hash. Matteo: if you have 10 brokers, you should have 100 partitions if you’re looking to spread the load across all of the brokers. Heesung: note that there are different splitting algorithms. Michael: there is not merging topic bundles, which makes splitting a “forever” event right now. Matteo: merging bundles was on the road map (like 8 years ago). It is doable, but it is complicated because it requires coordination between multiple nodes. In most cases, it’s not terrible to overshoot on the number of bundles. How can we support both scenarios, (Lari’s and the millions of topics scenario)? Lari: the design is limiting future improvements. Another use case from the mailing list related to PIP 192 broker graceful shutdown. Due to the way bundles are unloaded, there are interruptions for topic availability. If you could move topics atomically, there is a chance for a seamless handover. There could be an RPC between brokers to help smooth over issues, too. Matteo: that implies every broker knows about every topic. Lari: no, that isn’t my proposed design. Matteo: if you have ownership, then brokers need to know. In the work for 192, there is a concept of transfer. I wouldn’t want RPCs because it could become tricky with availability. Part of the PIP is to keep track of bundle placement using a topic. Since the brokers are aware of a transfer operation, broker A becomes aware that broker B is the new owner and a transfer needs to happen, which can enable a graceful transfer. Instead of cutting off the clients, you can close the bundle, transfer it, and then when B is ready, tell the clients where to connect to. I am not against getting rid of bundles, but how do we treat the case of millions of topics efficiently? Lari: the current metadata store sends all notifications to all brokers, so we need to add sharding so that not every broker gets every notification. Matteo: other part about the bundles is that in order to improve service discovery with many topic partitions or even just many topics, the lookups are individual instead of batched. This could be added to the protocol. It could be a waste of resources to do all of the lookups. The one downside is that it pushes the bundle mechanism to the client side, which is tight coupling. Michael: there are extra connections related to the many lookups too. Lari: one concept is that in str
[GitHub] [pulsar] michaeljmarshall added a comment to the discussion: Unsubscribe subscription [Rest API] working?
GitHub user michaeljmarshall added a comment to the discussion: Unsubscribe
subscription [Rest API] working?
It looks like that documentation is incorrect. I also get a 404 when I call
that endpoint. However, when I run `bin/pulsar-admin topics unsubscribe -s sub
test`, I noticed that the broker logs a different endpoint:
> 23:49:23.569 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO
> org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [01/Sep/2022:23:49:23
> -0500] "PUT /admin/v2/persistent/public/default/test/subscription/sub
> HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.7.2" 78
The correct endpoint includes `/persistent`. If you try
`/admin/v2/persistent/{tenant}/{namespace}/{topic}/subscription/{subName}`, it
should work.
Would you mind creating an issue in the project to get the documentation fixed?
Thanks!
GitHub link:
https://github.com/apache/pulsar/discussions/17379#discussioncomment-3532868
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] michaeljmarshall added a comment to the discussion: Unsubscribe subscription [Rest API] working?
GitHub user michaeljmarshall added a comment to the discussion: Unsubscribe subscription [Rest API] working? Thanks, that helps a ton. GitHub link: https://github.com/apache/pulsar/discussions/17379#discussioncomment-3532862 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[DISCUSS] Remove timestamp from Prometheus metrics
Hi Pulsar Community, Recently, we noticed in certain Grafana metrics from the broker that it appeared a topic had metrics reported by two different brokers at the same time. It turns out that the root of the problem is a concept called "staleness" in prometheus and it is directly related to the fact that we export timestamps with our metrics. As such, I wrote a PR to remove these timestamps [0]. In it, I propose that we remove the timestamps and cherry pick this fix to all active branches of Pulsar. The PR has more detail, so please see it if you're interested. If removing these timestamps will break your use case, please let me know. By my reading, we do not qualify as an application that needs to report timestamps. Additionally, I tried to make it configurable, but many of these classes are static, so it would be non-trivial to make the behavior configurable. Thanks, Michael [0] https://github.com/apache/pulsar/pull/17419
[RESULT][VOTE] PIP-200 Package Pulsar Trino distro and config in a dedicated folder
Hi, Thanks to everyone who participated in the vote. The voting is now closed and it has passed with 4 binding +1, 5 non-binding +1, and no 0 or -1 votes. +1 bindings: Penghui Li Enrico Olivelli Lari Hotari Jiwei Guo +1 non-bindings: Nicolò Boschi Max Xu Zixuan Liu Aloys Zhang tison Best, tison.
Re: [VOTE] PIP-200 Package Pulsar Trino distro and config in a dedicated folder
+1 (non-binding) Best, tison. tison 于2022年9月2日周五 09:46写道： > Thanks for your participation! I'm closing this vote now and summarize the > result. > > You're welcome to help with reviewing > https://github.com/apache/pulsar/pull/17062 and once it gets merged, I > will start to update the corresponding document and add a caveat. Then this > PIP is completed, easy ;-) > > Best, > tison. > > > Aloys Zhang 于2022年9月1日周四 13:53写道： > >> +1(non-binding) >> >> guo jiwei 于2022年9月1日周四 10:31写道： >> >> > +1 (binding) >> > >> > >> > Regards >> > Jiwei Guo (Tboy) >> > >> > >> > On Thu, Sep 1, 2022 at 1:36 AM Lari Hotari wrote: >> > >> > > +1 (binding) >> > > >> > > -Lari >> > > >> > > On 2022/08/29 13:25:51 tison wrote: >> > > > Hi devs, >> > > > >> > > > This is the official thread VOTE for PIP-200 Package Pulsar Trino >> > distro >> > > > and config in a dedicated folder. >> > > > >> > > > Here is the PIP issue: >> https://github.com/apache/pulsar/issues/17137 >> > > > Here is the discussion thread: >> > > > https://lists.apache.org/thread/s985ypf0r0hzcm0mx653n5h2rt7n273v >> > > > Here is the link to the draft PR: >> > > > https://github.com/apache/pulsar/pull/17062 >> > > > >> > > > Voting will stay open for at least 72h. >> > > > >> > > > Best, >> > > > tison. >> > > > >> > > >> > >> >
Re: [VOTE] PIP-200 Package Pulsar Trino distro and config in a dedicated folder
Thanks for your participation! I'm closing this vote now and summarize the result. You're welcome to help with reviewing https://github.com/apache/pulsar/pull/17062 and once it gets merged, I will start to update the corresponding document and add a caveat. Then this PIP is completed, easy ;-) Best, tison. Aloys Zhang 于2022年9月1日周四 13:53写道： > +1(non-binding) > > guo jiwei 于2022年9月1日周四 10:31写道： > > > +1 (binding) > > > > > > Regards > > Jiwei Guo (Tboy) > > > > > > On Thu, Sep 1, 2022 at 1:36 AM Lari Hotari wrote: > > > > > +1 (binding) > > > > > > -Lari > > > > > > On 2022/08/29 13:25:51 tison wrote: > > > > Hi devs, > > > > > > > > This is the official thread VOTE for PIP-200 Package Pulsar Trino > > distro > > > > and config in a dedicated folder. > > > > > > > > Here is the PIP issue: https://github.com/apache/pulsar/issues/17137 > > > > Here is the discussion thread: > > > > https://lists.apache.org/thread/s985ypf0r0hzcm0mx653n5h2rt7n273v > > > > Here is the link to the draft PR: > > > > https://github.com/apache/pulsar/pull/17062 > > > > > > > > Voting will stay open for at least 72h. > > > > > > > > Best, > > > > tison. > > > > > > > > > >
[GitHub] [pulsar] trevordowdle added a comment to the discussion: Unsubscribe subscription [Rest API] working?
GitHub user trevordowdle added a comment to the discussion: Unsubscribe subscription [Rest API] working? Sorry here is the link to my reply. https://github.com/apache/pulsar/discussions/17379#discussioncomment-3529738 GitHub link: https://github.com/apache/pulsar/discussions/17379#discussioncomment-3529746 This is an automatically sent email for dev@pulsar.apache.org. To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
[GitHub] [pulsar] trevordowdle added a comment to the discussion: Unsubscribe subscription [Rest API] working?
GitHub user trevordowdle added a comment to the discussion: Unsubscribe
subscription [Rest API] working?
Thank you for your response.
I am using the rest api, here is a sample of the end point:
"DELETE" request to:
http://pulsar-test.com:8080/admin/v2/namespaces/test/ingest/test/subscription/subscription-message-save
Result:
statusText = 'Not Found'
status = 404
Running: pulsar-admin topics stats persistent://test/ingest/test show the
subscription still active.
```
"subscriptions" : {
"subscription-message-save" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chuckedMessageRate" : 0,
"msgBacklog" : 0,
"backlogSize" : 0,
"msgBacklogNoDelayed" : 0,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"type" : "Exclusive",
"msgRateExpired" : 0.0,
"totalMsgExpired" : 0,
"lastExpireTimestamp" : 0,
"lastConsumedFlowTimestamp" : 1662056861336,
"lastConsumedTimestamp" : 0,
"lastAckedTimestamp" : 0,
"lastMarkDeleteAdvancedTimestamp" : 0,
"consumers" : [ ],
"isDurable" : true,
"isReplicated" : false,
"consumersAfterMarkDeletePosition" : { },
"nonContiguousDeletedMessagesRanges" : 0,
"nonContiguousDeletedMessagesRangesSerializedSize" : 0
}
},
```
What works:
"Post" request to
http://pulsar-test.com:8080/admin/v2/namespaces/test/ingest/unsubscribe/subscription-message-save
Result:
statusText = 'No Content'
status = 204
Running: pulsar-admin topics stats persistent://test/ingest/test shows the
subscription has been successfully removed.
`"subscriptions" : { },`
GitHub link:
https://github.com/apache/pulsar/discussions/17379#discussioncomment-3529738
This is an automatically sent email for dev@pulsar.apache.org.
To unsubscribe, please send an email to: dev-unsubscr...@pulsar.apache.org
Re: [DISCUSS] Call to improve Pulsar contributor's experience
On 2022/09/01 08:36:11 Yu wrote: > # 1 > For pure doc PRs (only update .md files), do they run the same tests as > code PRs? > If so, can we set them to run only doc-related tests and skip code tests > (since they're easily failed)? > In this way, docs can be iterated faster. The solution is already in place where the CI pipeline for docs is expedited. Some technical details about the solution: All builds steps in the GitHub Actions workflow build jobs are skipped for PRs that include changes only to docs. The reason why the workflows and build jobs aren't completely skipped is that we use the "required checks" feature and it is necessary to run all required checks also for PRs with only doc changes. > > > > # 2 > Does it make sense to add instructions for tests to the Pulsar Contribution > Guide? > > For example, > > * For users: > - How to resolve test issues (common test failure reasons and solutions) > - Who can ask for help if users are blocked and can not resolve problems > themselves > - How to report test bugs > > * For developers: > - How do tests work? (mechanism, Apache rules, etc) > - How can I add/update tests? (quotas [1], limitations, notes, etc) Good suggestions. In general, I hope we find better ways to listen to the voice of our contributors. What is their contribution experience? How did they feel about it? Perhaps we could decide to conduct surveys? GitHub discussions has support for polls [1] so that is one option as a technical solution for asking for feedback in a way where there would be a low barrier to respond. -Lari [1] https://github.blog/changelog/2022-04-12-discussions-polls/
Re: [DISCUSS] General question regarding PIP comments
Asaf, good question! Il giorno gio 1 set 2022 alle ore 15:01 Asaf Mesika ha scritto: > > Hi, > > I'm fairly new to this project and I have seen 2 options to comment to a > PIP: > 1. In the GitHub issue. > 2. In the mailing list. > > I was wondering what is the best practice. In Apache projects we generally prefer discussions on the mailing list, because it is easier to let the community participate in the discussions. Unfortunately in order to follow a discussion on GH you must actively follow the notifications (hundreds of messages per day) or use any other means to stay up to date. So we generally keep "discussions" here. When you go down to implementation details it is better to work on the PR, as it is more productive. But for high level discussion it is better that they happen here Discussing on the GH issue of the PIP is misleading, and I agree with you that we should keep the discussions only in one place Enrico > > I know there is an ongoing initiative at the moment to switch a PIP to > become a file and actually a PR for submitting it. I guess the question > also applies to this new scheme :) > > Thanks, > > Asaf
[DISCUSS] General question regarding PIP comments
Hi, I'm fairly new to this project and I have seen 2 options to comment to a PIP: 1. In the GitHub issue. 2. In the mailing list. I was wondering what is the best practice. I know there is an ongoing initiative at the moment to switch a PIP to become a file and actually a PR for submitting it. I guess the question also applies to this new scheme :) Thanks, Asaf
Re: [DISCUSS] Alternatives to changing public protocol
> > As Penghui suggested, this field name is changed to `message_id` for > potential generic usage. :) > > That's the thing - it's not really for potential generic use - it's more for potential *internal* generic usage, which is publicly exposed. When some outside visitor looks at the API and asks himself - "why should I provide a message ID for a message I'm publishing? Isn't ID something the broker creates for itself?" - This creates confusion, which leads IMO to less adoption and makes it harder to contribute. I'm quite new to Pulsar, and I feel that there is confusion in quite numerous parts of the system. My suggestion is raised here to try to avoid that confusion. > > The second problem is clients: Every such field will eventually trickle > > down to the clients, which will need to ignore that field. In my opinion, > > it makes it harder for the client's maintainers. Especially when the > > community goal is to expand and have many languages clients maintained by > > the community > > Our current client's implementation is quite complex already. Comparing > with this, > ignoring a few fields does not seems to be a significant hard thing in > this, > as long as we document it well, right? > > Having internal fields makes the client even more complex. It's not just about ignoring fields, it's about having more and more of them. What I suggest is separating to an internal API and internal client for those internal use cases. I'm not only referring to PIP-180, but to any future PIP. > > > I believe someone who tries to reason about Pulsar, and its architecture, > > by looking at its public API should not have any fields which will never > be > > relevant to the reader. It makes it hard to reason and understand the > > public API. > > > > This design principle of keeping the public API clean is clear and easy to > understand and I totally support this. But in the case of PIP-180 or > geo-replication, the replicator can be considered as a special producer > client, and it just inherited the basic semantic of a normal producer and > extended its abilities to support some special internal usage. > > Of course we can use a different protocol and different port for strictly > inter-broker communications in theory. But the side effect of this would be > more codes, more machine resource usage, harder to maintain, and longer > time to > make the feature steady, comparing with just extending the abilities of > producer client. > > If this come to a case that inter-broker communication is needed and it is > not > the case of producer or consumer, I think we should definitely consider to > introduce the dedicated port and protocols. > > Again, my suggestion mainly applies for the future - to make a conscious decision to avoid overloading more internal use cases to the public API. PIP-180 is currently a good case study to explore that suggestion (well, the ship has sailed, but it still is a good example). I reiterate what I said before: You can say your sentence for any new internal feature: "the X can be considered a special producer client , and it just inherited the basic semantic of a normal producer and extended its abilities to support some special internal usage". Replace X with any feature, thereby expanding the public API more and more with internal fields the normal user should never know about - the whole notion of encapsulation and simplicity. I would also like others to chime in on this and get their thoughts as well. > On 2022/07/20 15:47:16 Asaf Mesika wrote: > > Hi, > > > > We started discussing in PIP-180, which Penghui recommended I move to a > > dedicated thread. > > > > Pulsar has a public API in its binary protocol, which the clients use to > > communicate with it. Nonetheless, it is its public API to the server. > > > > I believe the public API should not be changed for internal communication > > purposes. PIP-180 gives a really good example: We would like to > introduce a > > new feature called Shadow Topic and would like to replicate messages from > > the source topic to the Shadow topic. It just so happens to be that the > > replication mechanism uses the Broker public API to send messages to a > > broker. The design would like to expand on that by adding a field to this > > public API, to serve that specific feature needs (the field is not > generic, > > it's specifically named shadow_message_id). > > > > I believe someone who tries to reason about Pulsar, and its architecture, > > by looking at its public API should not have any fields which will never > be > > relevant to the reader. It makes it hard to reason and understand the > > public API. > > > > The second problem is clients: Every such field will eventually trickle > > down to the clients, which will need to ignore that field. In my opinion, > > it makes it harder for the client's maintainers. Especially when the > > community goal is to expand and have many languages clients maintained by > > the community > > > > The
Re: [VOTE] [PIP-204] Extensions for BrokerInterceptor
+1 (non-binding) Best, Max Xu On Thu, Sep 1, 2022 at 12:12 PM tison wrote: > +1 (non-binding) > > Best, > tison. > > > PengHui Li 于2022年9月1日周四 11:20写道： > > > +1 (binding) > > > > Penghui > > > > On Thu, Sep 1, 2022 at 11:06 AM guo jiwei wrote: > > > > > +1 (binding) > > > > > > Regards > > > Jiwei Guo (Tboy) > > > > > > > > > On Wed, Aug 31, 2022 at 10:11 AM Haiting Jiang > > > > wrote: > > > > > > > +1 > > > > > > > > Thanks, > > > > Haiting > > > > > > > > On Tue, Aug 30, 2022 at 8:26 PM Enrico Olivelli > > > > > wrote: > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > Enrico > > > > > > > > > > Il giorno mar 30 ago 2022 alle ore 13:24 lin chen > > > > > ha scritto: > > > > > > > > > > > > +1 > > > > > > > > > > > > Aloys Zhang 于2022年8月30日周二 19:02写道： > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > This is the VOTE thread for PIP-204 Extensions for > > > > BrokerInterceptor. > > > > > > > > > > > > > > PIP issue: https://github.com/apache/pulsar/issues/17267 > > > > > > > > > > > > > > Discussion thread: > > > > > > > > https://lists.apache.org/thread/3zgpbxffo7gzsb5mdh0sk2rq27bgw2bo > > > > > > > > > > > > > > This voting will open for at least 72h. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > Aloys > > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] PIP-205: Reactive Java client for Apache Pulsar
+1 (non-binding) Best, Max Xu On Wed, Aug 31, 2022 at 10:06 PM Lari Hotari wrote: > Hello, > > This is the official VOTE thread for PIP-205: Reactive Java client for > Apache Pulsar. > PIP link: https://github.com/apache/pulsar/issues/17335 > discussion: > https://lists.apache.org/thread/xkfl5px7qgt5rrxw5pj0g318r6mbdlz1 > > Given the proposal is accepted, as the next step I will proceed to request > the creation of the proposed empty git repository > https://github.com/apache/pulsar-client-reactive and open it for > contributions. > > This vote is open for 72h and closes Sep-03-2022 at 15:30 UTC. > > BR, Lari >
Re: [VOTE] Pulsar Release 2.7.5 Candidate 3
Thank you all, Closing this Vote with 4 binding +1, and 2 non-binding +1. I will move forward with the release process. Thanks, Haiting On Thu, Sep 1, 2022 at 4:07 PM Enrico Olivelli wrote: > +1 (binding) > Built from sources and run some tests. > I wasn't able to run all the tests from pulsar-broker, it takes too > much time and there are flaky tests. > Btw we know the status of the branch, now for newer branches we are in > better shape. > There is no need to hold the release > > thank you Haiting > > Enrico > > Il giorno gio 1 set 2022 alle ore 09:51 PengHui Li > ha scritto: > > > > +1 (binding) > > > > - Checked the signatures > > - Started a standalone and use pulsar-perf to produce and consume > messages > > - Ran stateful function > > - Ran the connector > > > > Best, > > Penghui > > > > On Thu, Sep 1, 2022 at 2:52 PM guo jiwei wrote: > > > > > +1 (binding) > > > > > > - Verified checksum and signatures > > > - Apache Rat check passes > > > - Compiled from source > > > - Run Pulsar standalone > > > - Validate Pub/Sub and Java Functions > > > - Validate Stateful Functions > > > > > > Regards > > > Jiwei Guo (Tboy) > > > > > > > > > On Thu, Sep 1, 2022 at 11:36 AM Michael Marshall > > > > wrote: > > > > > > > +1 (binding) > > > > > > > > - Verified checksums and signatures on 45 artifacts > > > > - Compiled from source (apache-pulsar-2.7.5-src.tar.gz) > > > > using `mvn clean install -DskipTest` using JDK 8 > > > > - Ran `mvn apache-rat:check` successfully > > > > - Verified binary licenses > > > > - Verified that the lib directory jars from the binary tarball match > > > > my local build of the git tag (the only differences came from the > > > > pulsar jars, which is expected) > > > > - Ran pulsar standalone with the performance producer and consumer > using > > > > JDK 8 > > > > > > > > Thank you for managing the release, Haiting! > > > > > > > > Thanks, > > > > Michael > > > > > > > > > > > > On Wed, Aug 31, 2022 at 2:15 AM Haiting Jiang < > jianghait...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > > This is an interesting result. It would be great if the download > page > > > > > > after this release is made would indicate JDK versions. > > > > > > > > > > > > > > > Yes, this info seems missing. > > > > > I opened a PR[1] to add this info. Please take a look. > > > > > > > > > > [1] https://github.com/apache/pulsar/pull/17368 > > > > > > > > > > Thanks, > > > > > Haiting > > > > > > > > > > On Wed, Aug 31, 2022 at 2:31 PM Dave Fisher > > > > > wrote: > > > > > > > > > > > This is an interesting result. It would be great if the download > page > > > > > > after this release is made would indicate JDK versions. > > > > > > > > > > > > Thanks! > > > > > > Dave > > > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > > > On Aug 30, 2022, at 10:46 PM, Michael Marshall < > > > mmarsh...@apache.org > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Thank you both for verifying it on your end. After debugging > this > > > > for > > > > > > > a way too long tonight, I discovered my mistake was using Java > 11 > > > > > > > instead of Java 8. It didn't occur to me that the different > > > versions > > > > > > > could/would result in a different set of dependencies. > > > > > > > > > > > > > > When I use Java 8, this command passes now: > > > src/check-binary-license > > > > > > > distribution/server/target/apache-pulsar-2.7.5-bin.tar.gz. I > plan > > > to > > > > > > > finish my release validation tomorrow. > > > > > > > > > > > > > > Thanks! > > > > > > > Michael > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Aug 30, 2022 at 4:21 AM Anon Hxy > > > > > wrote: > > > > > > >> > > > > > > >> Hi Michael, > > > > > > >> > > > > > > >> I also couldn't reproduce it with the same steps Haiting > > > provided. > > > > > > >> > > > > > > >> Here is my local environment info with `mvn -v` > > > > > > >> > > > > > > >> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) > > > > > > >> Maven home: /Users/didi/Documents/apache-maven-3.6.3 > > > > > > >> Java version: 1.8.0_291, vendor: Oracle Corporation, runtime: > > > > > > >> > > > /Library/Java/JavaVirtualMachines/jdk1.8.0_291.jdk/Contents/Home/jre > > > > > > >> Default locale: zh_CN, platform encoding: UTF-8 > > > > > > >> OS name: "mac os x", version: "10.16", arch: "x86_64", family: > > > "mac" > > > > > > >> > > > > > > >> Thanks, > > > > > > >> Xiaoyu Hou > > > > > > >> > > > > > > >> > > > > > > >> Haiting Jiang 于2022年8月30日周二 15:23写道： > > > > > > >> > > > > > > >>> Hi Michael, > > > > > > >>> Thanks for your verification. > > > > > > >>> > > > > > > >>> I tried to recreate the issue, but I can't reproduce this. > > > > > > >>> Here is my steps: > > > > > > >>> - wget > > > > > > >>> > > > > > > >>> > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.5-candidate-3/apache-pulsar-2.7.5-src.tar.gz > > > > > > >>> - tar -xvf apache-pulsar-2.7.5-src.tar.gz > > > >
[GitHub] [pulsar-client-node] equanz opened a new pull request, #229: Update C++ client version and compatiblity table to v2.10.1
equanz opened a new pull request, #229: URL: https://github.com/apache/pulsar-client-node/pull/229 ### Motivation Relates to https://github.com/apache/pulsar-client-node/pull/215 and https://github.com/apache/pulsar/pull/15508 According to https://github.com/apache/pulsar-client-node/pull/215 , "double callback segfault" issue was fixed in https://github.com/apache/pulsar/pull/15508 . The https://github.com/apache/pulsar/pull/15508 fix was released as v2.10.1 ( https://github.com/apache/pulsar/commit/fa05a850e7356b42591fc04506496026c96ad878 ). We should use the fixed version of C++ client. ### Modifications * Modify `pulsar-version.txt` to use C++ client version to v2.10.1 in CI * Update compatibility table ### Verifying this change - [ ] Make sure that the change passes the CI checks. This change is a trivial rework / code cleanup without any test coverage. ### Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API: (no) - The schema: (no) - The default values of configurations: (no) - The wire protocol: (no) ### Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [pulsar-site] Anonymitaet commented on pull request #176: Generate client config docs from source code
Anonymitaet commented on PR #176: URL: https://github.com/apache/pulsar-site/pull/176#issuecomment-1233989322 @michaeljmarshall > In my opinion, these next docs are confusing to users since they document features that are not yet released. > I don't really think we should be exposing the unreleased configuration documentation for the current master branch on the pulsar.apache.org website. I think it's OK to show the `latest` docs. Reasons: 1. It may not cause confusion since many projects follow this way (as below). Users get used to it and they know the meaning of `latest`. Examples: - https://chaos-mesh.org/docs/next/ - https://cassandra.apache.org/doc/latest/ - https://docs.taichi-lang.org/docs/master/ - https://skywalking.apache.org/docs/main/latest/en/concepts-and-designs/overview/ 2. Pulsar documentation and website iterate fast and those changes are shown on `latest`. It's a good chance for users to capture fresh updates and for maintainers to promote new changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [DISCUSS] Call to improve Pulsar contributor's experience
Hi Lari, Thanks for raising this up! May I know your thoughts on these questions? # 1 For pure doc PRs (only update .md files), do they run the same tests as code PRs? If so, can we set them to run only doc-related tests and skip code tests (since they're easily failed)? In this way, docs can be iterated faster. # 2 Does it make sense to add instructions for tests to the Pulsar Contribution Guide? For example, * For users: - How to resolve test issues (common test failure reasons and solutions) - Who can ask for help if users are blocked and can not resolve problems themselves - How to report test bugs * For developers: - How do tests work? (mechanism, Apache rules, etc) - How can I add/update tests? (quotas [1], limitations, notes, etc) [1] https://github.com/apache/pulsar/issues/16439#issuecomment-1182832309 Thank you! On Thu, Sep 1, 2022 at 4:11 PM Lari Hotari wrote: > Hi, > > I think that we would need to improve the experience for contributors. > It's currently a big challenge to get a PR to the state where tests pass, > mainly because of the large amount of flaky tests and frequent congestions > in Pulsar CI. We don't tell this to the contributors in the PR template [1] > or the contributors guide [2] and finding this out without anyone > explaining could be a frustrating experience. > > Let's improve our contributor experience. "The hard part isn't solving the > problems, it's identifying the right problems to solve." [3] > > Would someone like to share their Pulsar contribution experience and what > you think that needs improvement? What was painful? > > -Lari > > [1] > https://raw.githubusercontent.com/apache/pulsar/master/.github/PULL_REQUEST_TEMPLATE.md > [2] https://pulsar.apache.org/contributing/ > [3] https://www.youtube.com/watch?v=qqaOpSJKdWc , > https://leanpub.com/ideaflow . Janelle Arty Starr: "Idea Flow: How to > Measure the PAIN in Software Development" >
Re: [DISCUSS] Call to improve Pulsar contributor's experience
I'm sharing my experience about finding the reason why the build fails. This has been frustrating. In the past, it has been a challenge to find the reason why the build fails. Not that we don't have the information about the failure, but the challenge is that there's too much information. :) Loading large log files in the GitHub Actions UI usually freezes the browser for some time (10-30 seconds), and it's slow and frustrating to find what caused the build failure. There's an improvement for this now. I improved finding the build failure recently in Pulsar CI. This guide explains how you can view Pulsar CI test failures in a simple and usable way: https://gist.github.com/lhotari/a76feddb9e77f2d316785614cb0a52dd (I'm planning to contribute this to the contributors guide eventually) -Lari
[DISCUSS] Call to improve Pulsar contributor's experience
Hi, I think that we would need to improve the experience for contributors. It's currently a big challenge to get a PR to the state where tests pass, mainly because of the large amount of flaky tests and frequent congestions in Pulsar CI. We don't tell this to the contributors in the PR template [1] or the contributors guide [2] and finding this out without anyone explaining could be a frustrating experience. Let's improve our contributor experience. "The hard part isn't solving the problems, it's identifying the right problems to solve." [3] Would someone like to share their Pulsar contribution experience and what you think that needs improvement? What was painful? -Lari [1] https://raw.githubusercontent.com/apache/pulsar/master/.github/PULL_REQUEST_TEMPLATE.md [2] https://pulsar.apache.org/contributing/ [3] https://www.youtube.com/watch?v=qqaOpSJKdWc , https://leanpub.com/ideaflow . Janelle Arty Starr: "Idea Flow: How to Measure the PAIN in Software Development"
Re: [VOTE] Pulsar Release 2.7.5 Candidate 3
+1 (binding) Built from sources and run some tests. I wasn't able to run all the tests from pulsar-broker, it takes too much time and there are flaky tests. Btw we know the status of the branch, now for newer branches we are in better shape. There is no need to hold the release thank you Haiting Enrico Il giorno gio 1 set 2022 alle ore 09:51 PengHui Li ha scritto: > > +1 (binding) > > - Checked the signatures > - Started a standalone and use pulsar-perf to produce and consume messages > - Ran stateful function > - Ran the connector > > Best, > Penghui > > On Thu, Sep 1, 2022 at 2:52 PM guo jiwei wrote: > > > +1 (binding) > > > > - Verified checksum and signatures > > - Apache Rat check passes > > - Compiled from source > > - Run Pulsar standalone > > - Validate Pub/Sub and Java Functions > > - Validate Stateful Functions > > > > Regards > > Jiwei Guo (Tboy) > > > > > > On Thu, Sep 1, 2022 at 11:36 AM Michael Marshall > > wrote: > > > > > +1 (binding) > > > > > > - Verified checksums and signatures on 45 artifacts > > > - Compiled from source (apache-pulsar-2.7.5-src.tar.gz) > > > using `mvn clean install -DskipTest` using JDK 8 > > > - Ran `mvn apache-rat:check` successfully > > > - Verified binary licenses > > > - Verified that the lib directory jars from the binary tarball match > > > my local build of the git tag (the only differences came from the > > > pulsar jars, which is expected) > > > - Ran pulsar standalone with the performance producer and consumer using > > > JDK 8 > > > > > > Thank you for managing the release, Haiting! > > > > > > Thanks, > > > Michael > > > > > > > > > On Wed, Aug 31, 2022 at 2:15 AM Haiting Jiang > > > wrote: > > > > > > > > > > > > > > This is an interesting result. It would be great if the download page > > > > > after this release is made would indicate JDK versions. > > > > > > > > > > > > Yes, this info seems missing. > > > > I opened a PR[1] to add this info. Please take a look. > > > > > > > > [1] https://github.com/apache/pulsar/pull/17368 > > > > > > > > Thanks, > > > > Haiting > > > > > > > > On Wed, Aug 31, 2022 at 2:31 PM Dave Fisher > > > wrote: > > > > > > > > > This is an interesting result. It would be great if the download page > > > > > after this release is made would indicate JDK versions. > > > > > > > > > > Thanks! > > > > > Dave > > > > > > > > > > Sent from my iPhone > > > > > > > > > > > On Aug 30, 2022, at 10:46 PM, Michael Marshall < > > mmarsh...@apache.org > > > > > > > > > wrote: > > > > > > > > > > > > Thank you both for verifying it on your end. After debugging this > > > for > > > > > > a way too long tonight, I discovered my mistake was using Java 11 > > > > > > instead of Java 8. It didn't occur to me that the different > > versions > > > > > > could/would result in a different set of dependencies. > > > > > > > > > > > > When I use Java 8, this command passes now: > > src/check-binary-license > > > > > > distribution/server/target/apache-pulsar-2.7.5-bin.tar.gz. I plan > > to > > > > > > finish my release validation tomorrow. > > > > > > > > > > > > Thanks! > > > > > > Michael > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Aug 30, 2022 at 4:21 AM Anon Hxy > > > wrote: > > > > > >> > > > > > >> Hi Michael, > > > > > >> > > > > > >> I also couldn't reproduce it with the same steps Haiting > > provided. > > > > > >> > > > > > >> Here is my local environment info with `mvn -v` > > > > > >> > > > > > >> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) > > > > > >> Maven home: /Users/didi/Documents/apache-maven-3.6.3 > > > > > >> Java version: 1.8.0_291, vendor: Oracle Corporation, runtime: > > > > > >> > > /Library/Java/JavaVirtualMachines/jdk1.8.0_291.jdk/Contents/Home/jre > > > > > >> Default locale: zh_CN, platform encoding: UTF-8 > > > > > >> OS name: "mac os x", version: "10.16", arch: "x86_64", family: > > "mac" > > > > > >> > > > > > >> Thanks, > > > > > >> Xiaoyu Hou > > > > > >> > > > > > >> > > > > > >> Haiting Jiang 于2022年8月30日周二 15:23写道： > > > > > >> > > > > > >>> Hi Michael, > > > > > >>> Thanks for your verification. > > > > > >>> > > > > > >>> I tried to recreate the issue, but I can't reproduce this. > > > > > >>> Here is my steps: > > > > > >>> - wget > > > > > >>> > > > > > >>> > > > > > > > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.5-candidate-3/apache-pulsar-2.7.5-src.tar.gz > > > > > >>> - tar -xvf apache-pulsar-2.7.5-src.tar.gz > > > > > >>> - cd apache-pulsar-2.7.5 > > > > > >>> - mvn clean install -DskipTests > > > > > >>> - src/check-binary-license > > > > > >>> distribution/server/target/apache-pulsar-2.7.5-bin.tar.gz > > > > > >>> > > > > > >>> And nothing appears in `check-binary-license` output. > > > > > >>> > > > > > >>> Here is my local environment info with `mvn -v` > > > > > >>> > > > > > >>> Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; > > > > > >>> 2019-04-05T03:00:29+08:00) > > > > > >>> Maven home: /Users/jason/apache-maven-3.6.1 > >
Re: [VOTE] Pulsar Release 2.7.5 Candidate 3
+1 (binding) - Checked the signatures - Started a standalone and use pulsar-perf to produce and consume messages - Ran stateful function - Ran the connector Best, Penghui On Thu, Sep 1, 2022 at 2:52 PM guo jiwei wrote: > +1 (binding) > > - Verified checksum and signatures > - Apache Rat check passes > - Compiled from source > - Run Pulsar standalone > - Validate Pub/Sub and Java Functions > - Validate Stateful Functions > > Regards > Jiwei Guo (Tboy) > > > On Thu, Sep 1, 2022 at 11:36 AM Michael Marshall > wrote: > > > +1 (binding) > > > > - Verified checksums and signatures on 45 artifacts > > - Compiled from source (apache-pulsar-2.7.5-src.tar.gz) > > using `mvn clean install -DskipTest` using JDK 8 > > - Ran `mvn apache-rat:check` successfully > > - Verified binary licenses > > - Verified that the lib directory jars from the binary tarball match > > my local build of the git tag (the only differences came from the > > pulsar jars, which is expected) > > - Ran pulsar standalone with the performance producer and consumer using > > JDK 8 > > > > Thank you for managing the release, Haiting! > > > > Thanks, > > Michael > > > > > > On Wed, Aug 31, 2022 at 2:15 AM Haiting Jiang > > wrote: > > > > > > > > > > > This is an interesting result. It would be great if the download page > > > > after this release is made would indicate JDK versions. > > > > > > > > > Yes, this info seems missing. > > > I opened a PR[1] to add this info. Please take a look. > > > > > > [1] https://github.com/apache/pulsar/pull/17368 > > > > > > Thanks, > > > Haiting > > > > > > On Wed, Aug 31, 2022 at 2:31 PM Dave Fisher > > wrote: > > > > > > > This is an interesting result. It would be great if the download page > > > > after this release is made would indicate JDK versions. > > > > > > > > Thanks! > > > > Dave > > > > > > > > Sent from my iPhone > > > > > > > > > On Aug 30, 2022, at 10:46 PM, Michael Marshall < > mmarsh...@apache.org > > > > > > > wrote: > > > > > > > > > > Thank you both for verifying it on your end. After debugging this > > for > > > > > a way too long tonight, I discovered my mistake was using Java 11 > > > > > instead of Java 8. It didn't occur to me that the different > versions > > > > > could/would result in a different set of dependencies. > > > > > > > > > > When I use Java 8, this command passes now: > src/check-binary-license > > > > > distribution/server/target/apache-pulsar-2.7.5-bin.tar.gz. I plan > to > > > > > finish my release validation tomorrow. > > > > > > > > > > Thanks! > > > > > Michael > > > > > > > > > > > > > > > > > > > >> On Tue, Aug 30, 2022 at 4:21 AM Anon Hxy > > wrote: > > > > >> > > > > >> Hi Michael, > > > > >> > > > > >> I also couldn't reproduce it with the same steps Haiting > provided. > > > > >> > > > > >> Here is my local environment info with `mvn -v` > > > > >> > > > > >> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f) > > > > >> Maven home: /Users/didi/Documents/apache-maven-3.6.3 > > > > >> Java version: 1.8.0_291, vendor: Oracle Corporation, runtime: > > > > >> > /Library/Java/JavaVirtualMachines/jdk1.8.0_291.jdk/Contents/Home/jre > > > > >> Default locale: zh_CN, platform encoding: UTF-8 > > > > >> OS name: "mac os x", version: "10.16", arch: "x86_64", family: > "mac" > > > > >> > > > > >> Thanks, > > > > >> Xiaoyu Hou > > > > >> > > > > >> > > > > >> Haiting Jiang 于2022年8月30日周二 15:23写道： > > > > >> > > > > >>> Hi Michael, > > > > >>> Thanks for your verification. > > > > >>> > > > > >>> I tried to recreate the issue, but I can't reproduce this. > > > > >>> Here is my steps: > > > > >>> - wget > > > > >>> > > > > >>> > > > > > > > https://dist.apache.org/repos/dist/dev/pulsar/pulsar-2.7.5-candidate-3/apache-pulsar-2.7.5-src.tar.gz > > > > >>> - tar -xvf apache-pulsar-2.7.5-src.tar.gz > > > > >>> - cd apache-pulsar-2.7.5 > > > > >>> - mvn clean install -DskipTests > > > > >>> - src/check-binary-license > > > > >>> distribution/server/target/apache-pulsar-2.7.5-bin.tar.gz > > > > >>> > > > > >>> And nothing appears in `check-binary-license` output. > > > > >>> > > > > >>> Here is my local environment info with `mvn -v` > > > > >>> > > > > >>> Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; > > > > >>> 2019-04-05T03:00:29+08:00) > > > > >>> Maven home: /Users/jason/apache-maven-3.6.1 > > > > >>> Java version: 1.8.0_291, vendor: Oracle Corporation, runtime: > > > > >>> > > /Library/Java/JavaVirtualMachines/jdk1.8.0_291.jdk/Contents/Home/jre > > > > >>> Default locale: en_CN, platform encoding: UTF-8 > > > > >>> OS name: "mac os x", version: "10.15.7", arch: "x86_64", family: > > "mac" > > > > >>> > > > > >>> Please check the settings again and I also will ask others to > help > > > > verify > > > > >>> the issue you mention. > > > > >>> > > > > >>> Thanks, > > > > >>> Haiting > > > > >>> > > > > >>> > > > > >>> On Tue, Aug 30, 2022 at 12:37 PM Michael Marshall < > > > > mmarsh...@apache.org> > > > > >>> wrote: > > > > >>> > > > > W
[GitHub] [pulsar-site] urfreespace commented on pull request #176: Generate client config docs from source code
urfreespace commented on PR #176: URL: https://github.com/apache/pulsar-site/pull/176#issuecomment-1233838116 > Yes, it's relevant for `website-next`, and can also generate docs for `website` as `WEBSITE` variable is controllable. I'm not sure whether the current configuration in the documentation is unreleased. @SignorMercurio in fact, @michaeljmarshall means the next version docs but not `website-next` directory, I think he means your configuration docs is generated based on the master branch, and the master branch is not a released branch, however @Anonymitaet how do you think? /cc @michaeljmarshall https://user-images.githubusercontent.com/76024046/187852454-f3c9d34a-f2fc-4595-9335-0184e489f56c.png";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
