[PR] [fix][io] Improve Kafka Connect source offset flushing logic [pulsar]

2025-08-20 Thread via GitHub
harangozop opened a new pull request, #24654: URL: https://github.com/apache/pulsar/pull/24654 Fixes #24646 ### Motivation When using Kafka Connect source connectors with transform filters that drop records (e.g., Debezium SMT Filter or Kafka Connect Fi

Re: [PR] [fix][broker] Fix race condition in MetadataStoreCacheLoader causing inconsistent availableBroker list caching [pulsar]

2025-08-20 Thread via GitHub
dao-jun merged PR #24639: URL: https://github.com/apache/pulsar/pull/24639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.ap

(pulsar) branch master updated: [fix][broker] Fix race condition in MetadataStoreCacheLoader causing inconsistent availableBroker list caching (#24639)

2025-08-20 Thread daojun
This is an automated email from the ASF dual-hosted git repository. daojun pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new d3c615c494d [fix][broker] Fix race condition in

Re: [I] [Bug] Wild Card subscription doesn't work for non-persistent topics [pulsar]

2025-08-20 Thread via GitHub
oneby-wang commented on issue #24310: URL: https://github.com/apache/pulsar/issues/24310#issuecomment-3209003368 Hi, I just happen to read this part source code. TopicsPattern only works for persistent topics, and is triggered by MetaStore notification. It seems non-persistent topic metadat

Re: [PR] [improve][broker] Implement PIP-430 Pulsar Broker cache improvements [pulsar]

2025-08-20 Thread via GitHub
lhotari commented on PR #24623: URL: https://github.com/apache/pulsar/pull/24623#issuecomment-3208984303 Interestingly, the results on my Dell XPS 2019 (i9-9980HK CPU @ 2.40GHz) are very different than on MBP M3 Max: ``` ┌┬──┬──┬

Re: [PR] [fix][broker] Remove consumer from cache if closed before creation [pulsar]

2025-08-20 Thread via GitHub
nodece closed pull request #24638: [fix][broker] Remove consumer from cache if closed before creation URL: https://github.com/apache/pulsar/pull/24638 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] [fix][broker] Remove consumer from cache if closed before creation [pulsar]

2025-08-20 Thread via GitHub
nodece commented on PR #24638: URL: https://github.com/apache/pulsar/pull/24638#issuecomment-3208882454 Thanks for the review and discussion. After re-evaluating the logic, I realized that the current behavior is correct: The consumer should only be removed from the cache once the subscript

Re: [PR] Change the pulsar_client_sending_buffers_count metric to client level [pulsar-client-go]

2025-08-20 Thread via GitHub
RobertIndie merged PR #1408: URL: https://github.com/apache/pulsar-client-go/pull/1408 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr.

(pulsar-client-go) branch master updated: Change the pulsar_client_sending_buffers_count metric to client level (#1408)

2025-08-20 Thread zike
This is an automated email from the ASF dual-hosted git repository. zike pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar-client-go.git The following commit(s) were added to refs/heads/master by this push: new d471a678 Change the pulsar_client_sendin

Re: [PR] Fix hasMessageAvailable incorrectly returns true when read to latest after seeking by timestamp [pulsar-client-cpp]

2025-08-20 Thread via GitHub
Copilot commented on code in PR #498: URL: https://github.com/apache/pulsar-client-cpp/pull/498#discussion_r2289724959 ## lib/ConsumerImpl.cc: ## @@ -1578,10 +1582,16 @@ void ConsumerImpl::hasMessageAvailableAsync(const HasMessageAvailableCallback& c { std::lock_g

(pulsar) branch master updated: [improve][pip]PIP-436: Add decryptFailListener to Consumer (#24572)

2025-08-20 Thread penghui
This is an automated email from the ASF dual-hosted git repository. penghui pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new cb56281b36f [improve][pip]PIP-436: Add decryptF

Re: [I] [Bug] Topic creation encountered an exception by initialize topic policies service. topic_name=persistent://Research/SubmissionsApi/DataEvents error_message={"errorMsg":"Error while reading l

2025-08-20 Thread via GitHub
JingTyagi-ICON commented on issue #24553: URL: https://github.com/apache/pulsar/issues/24553#issuecomment-3207019792 Thanks @lhotari for your information. I wasn't able to work on till now due to other priorities. I was upgrading from chart version 3.8.0/app version 4.0.1 to chart version

[PR] Update Getting Started documentation with latest requirements and best practices [pulsar-site]

2025-08-20 Thread via GitHub
codelipenghui opened a new pull request, #1033: URL: https://github.com/apache/pulsar-site/pull/1033 ## Summary - Update Getting Started documentation with comprehensive prerequisites and latest version requirements - Enhance all deployment methods (standalone, Docker, Docker Compose,

Re: [PR] [improve][pip] PIP-437: Granular Control for Message Delivery Delays [pulsar]

2025-08-20 Thread via GitHub
chrisqwang3 commented on code in PR #24625: URL: https://github.com/apache/pulsar/pull/24625#discussion_r2288989546 ## pip/pip-437.md: ## @@ -0,0 +1,134 @@ +# PIP-437: Granular and Fixed-Delay Policies for Message Delivery + +# Background +Pulsar's delayed delivery feature allow

Re: [PR] [improve][broker] Implement PIP-430 Pulsar Broker cache improvements [pulsar]

2025-08-20 Thread via GitHub
lhotari commented on PR #24623: URL: https://github.com/apache/pulsar/pull/24623#issuecomment-3207584881 I fixed the test case and improved many things. I now limited the producer rate to 5 msg/s which doesn't show large differences other than with the current default Pulsar caching

Re: [PR] [improve][pip] Standardize PIP filenames and add comprehensive table of contents [pulsar]

2025-08-20 Thread via GitHub
codelipenghui commented on PR #24653: URL: https://github.com/apache/pulsar/pull/24653#issuecomment-3207538721 /pulsarbot run-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[PR] [improve][pip] Standardize PIP filenames and add comprehensive table of contents [pulsar]

2025-08-20 Thread via GitHub
codelipenghui opened a new pull request, #24653: URL: https://github.com/apache/pulsar/pull/24653 ## Summary This PR standardizes Pulsar Improvement Proposal (PIP) filenames and adds a comprehensive table of contents to improve organization and discoverability. ### Changes Made

Re: [PR] [improve][pip] Standardize PIP filenames and add comprehensive table of contents [pulsar]

2025-08-20 Thread via GitHub
github-actions[bot] commented on PR #24653: URL: https://github.com/apache/pulsar/pull/24653#issuecomment-3207537132 @codelipenghui Please add the following content to your PR description and select a checkbox: ``` - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [

(pulsar) branch master updated: [improve][client] Add OpenTelemetry metrics for client memory buffer usage (#24647)

2025-08-20 Thread penghui
This is an automated email from the ASF dual-hosted git repository. penghui pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new a66e8068058 [improve][client] Add OpenTelemetry

Re: [PR] [improve][pip]PIP-436: Add decryptFailListener to Consumer [pulsar]

2025-08-20 Thread via GitHub
codelipenghui merged PR #24572: URL: https://github.com/apache/pulsar/pull/24572 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pul

Re: [PR] [improve][client] Add OpenTelemetry metrics for client memory buffer usage [pulsar]

2025-08-20 Thread via GitHub
codelipenghui merged PR #24647: URL: https://github.com/apache/pulsar/pull/24647 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pul

Re: [PR] Add Java client OpenTelemetry metrics documentation [pulsar-site]

2025-08-20 Thread via GitHub
codelipenghui merged PR #1032: URL: https://github.com/apache/pulsar-site/pull/1032 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@

(pulsar-site) branch main updated: Add Java client OpenTelemetry metrics documentation (#1032)

2025-08-20 Thread penghui
This is an automated email from the ASF dual-hosted git repository. penghui pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 1776ba04479 Add Java client OpenTelemetry metr

Re: [I] [Bug] pulsar-io EventLoopGroup is used in dispatchers to execute synchronized code and CPU intensive work [pulsar]

2025-08-20 Thread via GitHub
lhotari commented on issue #23865: URL: https://github.com/apache/pulsar/issues/23865#issuecomment-3207059519 this is a real performance issue with Pulsar. In #24623, I have added a solution for profiling test cases with Async Profiler with JFR integrated to it. It's possible to analyze the

Re: [PR] [fix][broker] Remove consumer from cache if closed before creation [pulsar]

2025-08-20 Thread via GitHub
nodece commented on PR #24638: URL: https://github.com/apache/pulsar/pull/24638#issuecomment-3206919634 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

(pulsar) branch master updated: [fix][offload] Exclude unnecessary dependencies from tiered storage provider / offloader nar files (#24649)

2025-08-20 Thread lhotari
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 3c70e09af8f [fix][offload] Exclude unnecessary

Re: [PR] [fix][offload] Exclude unnecessary dependencies from tiered storage provider / offloader nar files [pulsar]

2025-08-20 Thread via GitHub
lhotari merged PR #24649: URL: https://github.com/apache/pulsar/pull/24649 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.ap

Re: [PR] [fix][offload] Exclude unnecessary dependencies from tiered storage provider / offloader nar files [pulsar]

2025-08-20 Thread via GitHub
codecov-commenter commented on PR #24649: URL: https://github.com/apache/pulsar/pull/24649#issuecomment-3206843308 ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/24649?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments

Re: [I] [Bug] Key_Shared or Shared subscription doesn't always deliver messages from the replay queue after a consumer disconnects and leaves a backlog unless new messages are produced [pulsar]

2025-08-20 Thread via GitHub
lhotari commented on issue #23845: URL: https://github.com/apache/pulsar/issues/23845#issuecomment-3206837535 @tarmacmonsterg It's possible that there's a remaining issue where the message delivery doesn't resume unless new messages are produced. It's what this current issue is about. Howev

Re: [I] [Bug] Key_Shared or Shared subscription doesn't always deliver messages from the replay queue after a consumer disconnects and leaves a backlog unless new messages are produced [pulsar]

2025-08-20 Thread via GitHub
lhotari commented on issue #23845: URL: https://github.com/apache/pulsar/issues/23845#issuecomment-3206821109 > We encountered a similar issue on version 4.0.6. Messages may not be delivered when adding or removing consumers (such as during restarts of our service pods). Messages get stuck

Re: [PR] [fix][sec] Upgrade bouncycastle bcpkix-fips version to 1.79 to address CVE-2025-8916 [pulsar]

2025-08-20 Thread via GitHub
lhotari merged PR #24650: URL: https://github.com/apache/pulsar/pull/24650 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.ap

(pulsar) branch master updated: [fix][sec] Upgrade bouncycastle bcpkix-fips version to 1.79 to address CVE-2025-8916 (#24650)

2025-08-20 Thread lhotari
This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new e78068a0430 [fix][sec] Upgrade bouncycastle bcp

Re: [PR] [fix][sec] Upgrade bouncycastle bcpkix-fips version to 1.79 to address CVE-2025-8916 [pulsar]

2025-08-20 Thread via GitHub
codecov-commenter commented on PR #24650: URL: https://github.com/apache/pulsar/pull/24650#issuecomment-3206725710 ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/24650?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments

[PR] [improve][broker] PIP-433: Ensure topic creation before starting GEO [pulsar]

2025-08-20 Thread via GitHub
nodece opened a new pull request, #24652: URL: https://github.com/apache/pulsar/pull/24652 PIP: #24485 ### Motivation In a GEO replication scenario, if the remote cluster does not have the replicated topic and the auto-creation type differs between the local and remote cluste

Re: [PR] [fix][offload] Exclude unnecessary dependencies from tiered storage provider / offloader nar files [pulsar]

2025-08-20 Thread via GitHub
lhotari commented on code in PR #24649: URL: https://github.com/apache/pulsar/pull/24649#discussion_r2288354449 ## tiered-storage/file-system/pom.xml: ## @@ -31,39 +31,44 @@ tiered-storage-file-system Apache Pulsar :: Tiered Storage :: File System - -

Re: [PR] Change the pulsar_client_sending_buffers_count metric to client level [pulsar-client-go]

2025-08-20 Thread via GitHub
Copilot commented on code in PR #1408: URL: https://github.com/apache/pulsar-client-go/pull/1408#discussion_r2288283257 ## pulsar/internal/buffer.go: ## @@ -124,8 +133,9 @@ type buffer struct { readerIdx uint32 writerIdx uint32 - refCnt atomic.Int64 -

Re: [PR] Change the pulsar_client_sending_buffers_count metric to client level [pulsar-client-go]

2025-08-20 Thread via GitHub
BewareMyPower commented on PR #1408: URL: https://github.com/apache/pulsar-client-go/pull/1408#issuecomment-3206510251 @crossoverJie I have updated the PR description, PTAL again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] [fix][offload] Exclude unnecessary dependencies from tiered storage provider / offloader nar files [pulsar]

2025-08-20 Thread via GitHub
yaalsn commented on code in PR #24649: URL: https://github.com/apache/pulsar/pull/24649#discussion_r2288248154 ## tiered-storage/file-system/pom.xml: ## @@ -31,39 +31,44 @@ tiered-storage-file-system Apache Pulsar :: Tiered Storage :: File System - -

Re: [PR] [fix][broker] Add double-check for non-durable cursor creation [pulsar]

2025-08-20 Thread via GitHub
nodece merged PR #24643: URL: https://github.com/apache/pulsar/pull/24643 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apa

(pulsar) branch master updated (34f86576758 -> e642e179804)

2025-08-20 Thread zixuan
This is an automated email from the ASF dual-hosted git repository. zixuan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git from 34f86576758 [fix][ws] Allow websocket principals to specify originalPrincipal without proxy role (#24642) add e6

[PR] [fix][broker]Failed to create partitions after the partitions were deleted because topic GC [pulsar]

2025-08-20 Thread via GitHub
poorbarcode opened a new pull request, #24651: URL: https://github.com/apache/pulsar/pull/24651 ### Motivation Configurations: - `allowAutoTopicCreation`: `false` - `brokerDeleteInactiveTopicsEnabled`: `true` - `brokerDeleteInactivePartitionedTopicMetadataEnabled`: `false`

[PR] [fix][sec] Upgrade bouncycastle version to 1.79 to address CVE-2025-8916 [pulsar]

2025-08-20 Thread via GitHub
lhotari opened a new pull request, #24650: URL: https://github.com/apache/pulsar/pull/24650 ### Motivation & Modifications Upgrade bouncycastle version to 1.79 to address CVE-2025-8916 ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-neede

[PR] [fix][offload] Exclude unnecessary dependencies from tiered storage provider / offloader nar files [pulsar]

2025-08-20 Thread via GitHub
lhotari opened a new pull request, #24649: URL: https://github.com/apache/pulsar/pull/24649 ### Motivation There are a lot of unnecessary dependencies packaged in tiered storage provider / offloader nar files. ### Modifications - make `managed-ledger` dependency a provid

Re: [PR] [improve][broker] Implement PIP-430 Pulsar Broker cache improvements [pulsar]

2025-08-20 Thread via GitHub
lhotari commented on PR #24623: URL: https://github.com/apache/pulsar/pull/24623#issuecomment-3206130960 The above results aren't correct, there was a flaw in the test. I'll publish new results. -- This is an automated message from the Apache Git Service. To respond to the message, please

[PR] [fix][broker]User topic failed to delete after removed cluster because of failed delete data from transaction buffer topic [pulsar]

2025-08-20 Thread via GitHub
poorbarcode opened a new pull request, #24648: URL: https://github.com/apache/pulsar/pull/24648 ### Motivation Background - Users may start 2 clusters with the Geo-Replication feature with Global ZK - Users want to hire one cluster, so remove the cluster from `namespace-lev

Re: [PR] [fix][broker] Remove consumer from cache if closed before creation [pulsar]

2025-08-20 Thread via GitHub
poorbarcode commented on code in PR #24638: URL: https://github.com/apache/pulsar/pull/24638#discussion_r2287806948 ## pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ServerCnxTest.java: ## @@ -2436,44 +2436,32 @@ public void testSubscribeBookieTimeout() throws Exc

[PR] Add Java client OpenTelemetry metrics documentation [pulsar-site]

2025-08-20 Thread via GitHub
codelipenghui opened a new pull request, #1032: URL: https://github.com/apache/pulsar-site/pull/1032 ## Summary This PR adds comprehensive documentation for all Java client OpenTelemetry metrics to the reference documentation (`docs/reference-metrics-opentelemetry.md`). ### Wh

Re: [PR] [fix][broker] Remove consumer from cache if closed before creation [pulsar]

2025-08-20 Thread via GitHub
nodece commented on code in PR #24638: URL: https://github.com/apache/pulsar/pull/24638#discussion_r2287774468 ## pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ServerCnxTest.java: ## @@ -2436,44 +2436,32 @@ public void testSubscribeBookieTimeout() throws Exceptio

Re: [I] [Bug] Key_Shared or Shared subscription doesn't always deliver messages from the replay queue after a consumer disconnects and leaves a backlog unless new messages are produced [pulsar]

2025-08-20 Thread via GitHub
tarmacmonsterg commented on issue #23845: URL: https://github.com/apache/pulsar/issues/23845#issuecomment-3205378482 We encountered a similar issue on version 4.0.6. Messages may not be delivered when adding or removing consumers (such as during restarts of our service pods). Messages get s

Re: [PR] [fix][broker] Add double-check for non-durable cursor creation [pulsar]

2025-08-20 Thread via GitHub
codecov-commenter commented on PR #24643: URL: https://github.com/apache/pulsar/pull/24643#issuecomment-3204845554 ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/24643?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments

Re: [PR] [fix][broker] Remove consumer from cache if closed before creation [pulsar]

2025-08-20 Thread via GitHub
poorbarcode commented on code in PR #24638: URL: https://github.com/apache/pulsar/pull/24638#discussion_r2287327123 ## pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ServerCnxTest.java: ## @@ -2436,44 +2436,32 @@ public void testSubscribeBookieTimeout() throws Exc

Re: [PR] [fix][broker] Remove consumer from cache if closed before creation [pulsar]

2025-08-20 Thread via GitHub
poorbarcode commented on code in PR #24638: URL: https://github.com/apache/pulsar/pull/24638#discussion_r2287327123 ## pulsar-broker/src/test/java/org/apache/pulsar/broker/service/ServerCnxTest.java: ## @@ -2436,44 +2436,32 @@ public void testSubscribeBookieTimeout() throws Exc

Re: [PR] [fix][broker] Add double-check for non-durable cursor creation [pulsar]

2025-08-20 Thread via GitHub
nodece commented on PR #24643: URL: https://github.com/apache/pulsar/pull/24643#issuecomment-3204473836 /pulsarbot rerun-failure-checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi