(pulsar) branch dependabot/maven/pulsar-io/solr/org.apache.solr-solr-core-8.11.3 deleted (was 1929569fb6a)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/pulsar-io/solr/org.apache.solr-solr-core-8.11.3 in repository https://gitbox.apache.org/repos/asf/pulsar.git was 1929569fb6a Bump org.apache.solr:solr-core from 8.11.1 to 8.11.3 in /pulsar-io/solr The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(pulsar) branch master updated: [fix] Bump org.apache.solr:solr-core from 8.11.1 to 8.11.3 in /pulsar-io/solr (#22047)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 7a90426253e [fix] Bump org.apache.solr:solr-core from 8.11.1 to 8.11.3 in /pulsar-io/solr (#22047) 7a90426253e is described below commit 7a90426253e96a995e5d3a254c76cb80a3d54c7b Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Fri Feb 9 16:58:41 2024 -0800 [fix] Bump org.apache.solr:solr-core from 8.11.1 to 8.11.3 in /pulsar-io/solr (#22047) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- pulsar-io/solr/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pulsar-io/solr/pom.xml b/pulsar-io/solr/pom.xml index ce56f1a0502..5be2639c718 100644 --- a/pulsar-io/solr/pom.xml +++ b/pulsar-io/solr/pom.xml @@ -29,7 +29,7 @@ -8.11.1 +8.11.3 pulsar-io-solr
Re: [PR] [fix] Bump org.apache.solr:solr-core from 8.11.1 to 8.11.3 in /pulsar-io/solr [pulsar]
codecov-commenter commented on PR #22047: URL: https://github.com/apache/pulsar/pull/22047#issuecomment-1936783061 ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22047?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Comparison is base [(`3036783`)](https://app.codecov.io/gh/apache/pulsar/commit/303678364eab538c16041214cae1588a5b2111d9?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) 36.56% compared to head [(`1929569`)](https://app.codecov.io/gh/apache/pulsar/pull/22047?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) 73.66%. > Report is 2 commits behind head on master. Additional details and impacted files [](https://app.codecov.io/gh/apache/pulsar/pull/22047?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) ```diff @@ Coverage Diff @@ ## master #22047 +/- ## = + Coverage 36.56% 73.66% +37.10% - Complexity1241832544+20126 = Files 1729 1865 +136 Lines132076 138968 +6892 Branches 1445215236 +784 = + Hits 48295 102373+54078 + Misses7738728686-48701 - Partials 6394 7909 +1515 ``` | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/22047/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/22047/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `24.43% <ø> (+0.29%)` | :arrow_up: | | [systests](https://app.codecov.io/gh/apache/pulsar/pull/22047/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `24.30% <ø> (+0.34%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/22047/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `72.95% <ø> (+40.97%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. [see 1443 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/22047/indirect-changes?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] Bump org.apache.solr:solr-core from 8.11.1 to 8.11.3 in /pulsar-io/solr [pulsar]
merlimat merged PR #22047: URL: https://github.com/apache/pulsar/pull/22047 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [feat][misc] PIP-320: Add OpenTelemetry scaffolding (#22010)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 8957e353ded [feat][misc] PIP-320: Add OpenTelemetry scaffolding (#22010) 8957e353ded is described below commit 8957e353ded9ee24eccea349c7747da721d9e66a Author: Dragos Misca AuthorDate: Fri Feb 9 15:40:20 2024 -0800 [feat][misc] PIP-320: Add OpenTelemetry scaffolding (#22010) Co-authored-by: Matteo Merli --- .github/workflows/pulsar-ci.yaml | 3 + build/run_integration_group.sh | 4 + distribution/server/src/assemble/LICENSE.bin.txt | 28 +++ pom.xml| 31 .../ProxySaslAuthenticationTest.java | 1 + pulsar-broker-common/pom.xml | 18 ++ .../stats/prometheus/PrometheusMetricsClient.java | 0 pulsar-broker/pom.xml | 14 ++ .../org/apache/pulsar/broker/PulsarService.java| 6 + .../broker/stats/PulsarBrokerOpenTelemetry.java| 49 + pulsar-functions/worker/pom.xml| 6 + .../worker/PulsarWorkerOpenTelemetry.java | 48 + .../functions/worker/PulsarWorkerService.java | 6 + .../worker/FunctionAssignmentTailerTest.java | 5 + .../pom.xml| 88 - .../opentelemetry/OpenTelemetryAttributes.java | 32 .../pulsar/opentelemetry/OpenTelemetryService.java | 108 +++ .../apache/pulsar/opentelemetry/package-info.java | 24 +++ .../opentelemetry/OpenTelemetryServiceTest.java| 201 + pulsar-proxy/pom.xml | 6 + .../apache/pulsar/proxy/server/ProxyService.java | 7 + .../proxy/stats/PulsarProxyOpenTelemetry.java | 49 + .../extensions/SimpleProxyExtensionTestBase.java | 1 + .../server/AdminProxyHandlerKeystoreTLSTest.java | 1 + .../proxy/server/AuthedAdminProxyHandlerTest.java | 1 + .../proxy/server/ProxyAdditionalServletTest.java | 1 + .../ProxyAuthenticatedProducerConsumerTest.java| 1 + .../proxy/server/ProxyAuthenticationTest.java | 4 +- .../server/ProxyConnectionThrottlingTest.java | 1 + .../server/ProxyEnableHAProxyProtocolTest.java | 1 + .../proxy/server/ProxyForwardAuthDataTest.java | 4 +- .../server/ProxyKeyStoreTlsTransportTest.java | 1 + .../proxy/server/ProxyKeyStoreTlsWithAuthTest.java | 1 + .../server/ProxyKeyStoreTlsWithoutAuthTest.java| 1 + .../proxy/server/ProxyLookupThrottlingTest.java| 1 + .../pulsar/proxy/server/ProxyMutualTlsTest.java| 1 + .../pulsar/proxy/server/ProxyParserTest.java | 1 + .../pulsar/proxy/server/ProxyRefreshAuthTest.java | 4 +- .../proxy/server/ProxyRolesEnforcementTest.java| 4 +- .../ProxyServiceStarterDisableZeroCopyTest.java| 3 +- .../proxy/server/ProxyServiceStarterTest.java | 1 + .../proxy/server/ProxyServiceTlsStarterTest.java | 1 + .../apache/pulsar/proxy/server/ProxyStatsTest.java | 1 + .../proxy/server/ProxyStuckConnectionTest.java | 1 + .../org/apache/pulsar/proxy/server/ProxyTest.java | 1 + .../apache/pulsar/proxy/server/ProxyTlsTest.java | 1 + .../pulsar/proxy/server/ProxyTlsWithAuthTest.java | 1 + .../server/ProxyWithAuthorizationNegTest.java | 4 +- .../proxy/server/ProxyWithAuthorizationTest.java | 5 +- .../server/ProxyWithExtensibleLoadManagerTest.java | 1 + .../server/ProxyWithJwtAuthorizationTest.java | 4 +- .../server/ProxyWithoutServiceDiscoveryTest.java | 5 +- .../SuperUserAuthedAdminProxyHandlerTest.java | 1 + .../server/UnauthedAdminProxyHandlerTest.java | 1 + tests/integration/pom.xml | 20 +- .../OpenTelemetryCollectorContainer.java | 63 +++ .../integration/containers/PulsarContainer.java| 2 + .../metrics/OpenTelemetrySanityTest.java | 165 + .../integration/topologies/PulsarCluster.java | 43 +++-- .../integration/topologies/PulsarClusterSpec.java | 17 ++ .../containers/otel-collector-config.yaml | 43 + .../src/test/resources/pulsar-metrics.xml | 28 +++ tests/integration/src/test/resources/pulsar.xml| 1 + 63 files changed, 1100 insertions(+), 76 deletions(-) diff --git a/.github/workflows/pulsar-ci.yaml b/.github/workflows/pulsar-ci.yaml index 7767beaa9aa..effeab90beb 100644 --- a/.github/workflows/pulsar-ci.yaml +++ b/.github/workflows/pulsar-ci.yaml @@ -589,6 +589,9 @@ jobs: - name: Transaction group: TRANSACTION + - name: Metrics +group: METRICS + steps: - name: checkout uses: actions/checkout@v4 diff --git a/build/run_integration_group.sh
Re: [PR] [feat][misc] PIP-320: Add OpenTelemetry scaffolding [pulsar]
merlimat merged PR #22010: URL: https://github.com/apache/pulsar/pull/22010 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [feat][misc] PIP-320: Add OpenTelemetry scaffolding [pulsar]
dragosvictor commented on code in PR #22010:
URL: https://github.com/apache/pulsar/pull/22010#discussion_r1484884025
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java:
##
@@ -897,6 +902,7 @@ public void start() throws PulsarServerException {
}
this.metricsGenerator = new MetricsGenerator(this);
+this.openTelemetry = new PulsarBrokerOpenTelemetry(config);
Review Comment:
High-level details about the no-op implementation can be found
[here](https://opentelemetry.io/docs/specs/otel/metrics/noop/).
For a more in-depth view, the auto-configured SDK instantiates an empty
`OpenTelemetrySdk` if telemetry is disabled
([ref](https://github.com/open-telemetry/opentelemetry-java/blob/ee6c9867d736af7359003e0b2f61ebf082e6fe3e/sdk-extensions/autoconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/AutoConfiguredOpenTelemetrySdkBuilder.java#L402)).
Since this object does not have any metric readers configured, it will use the
static no-op `MeterProvider`
([ref](https://github.com/open-telemetry/opentelemetry-java/blob/f421ef1e73906f6332f4a5ab39698b00efb4525c/sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/SdkMeterProvider.java#L96)),
which in turn relies on the no-op `MeterBuilder`
([ref](https://github.com/open-telemetry/opentelemetry-java/blob/695ed5350b2ef218c38087a627549f4de7b86c22/api/all/src/main/java/io/opentelemetry/api/metrics/DefaultMeterProvider.java#L16))
and finally the no-op `Meter` object
([ref](https://github.com/open-telemetry/opentelemetry-java/blob/695ed5
350b2ef218c38087a627549f4de7b86c22/api/all/src/main/java/io/opentelemetry/api/metrics/DefaultMeter.java#L19)).
In effect, the little memory resources that it uses even when disabled are a
handful of null fields and empty collections in `OpenTelemetrySdk` and its
children providers.
Regarding actual runtime usage, both CPU and memory, it will depend on how
we leverage the library. PIP-264 describes the overall plan:
- High-cardinality metrics shall be fetched via a batch callback, on top of
'asynchronous' meters. The benefit here is that we can iterate over an entire
collection (say, topics) just once and record each object's metrics in one
shot. This also lets us instantiate the respective `Attribute` array,
presumably different for each entity, just one time, thereby reducing memory
allocations. The associated cost is the meter itself, which must be built, but
it would reference the same underlying object when the library is disabled.
Since the callback is expected to perform the iteration, we don't need a meter
for each object, just one per object "class" (say, `TopicStats`); for instance,
if we assume that topics have 100 metrics associated, we will add 100 objects
to memory, independent of how many topics the broker is hosting. Each metric is
different, but our goal is to leverage these callbacks to pull the numbers from
existing data sources (`AtomicInteger`, `LongAdder`, etc.). Even be
tter, the callback does nothing
([ref](https://github.com/open-telemetry/opentelemetry-java/blob/695ed5350b2ef218c38087a627549f4de7b86c22/api/all/src/main/java/io/opentelemetry/api/metrics/DefaultMeter.java#L64))
if the library is disabled, so the compute overhead is effectively zero. This
should be our most common usage pattern.
- Histogram meters are 'synchronous' in OpenTelemetry jargon. This means we
must create an object for each entity type we want to monitor. If we want
namespace-level histograms for a metric, we will have to instantiate such a
histogram for each namespace. Because of this, the consensus in the PIP was to
eliminate such metrics from high-cardinality objects
([ref](https://github.com/apache/pulsar/blob/master/pip/pip-264.md#moving-topic-level-histograms-to-namespace-and-broker-level-only)).
This keeps memory usage under control. These histograms do nothing when the
library is disabled, so the CPU usage is practically zero again.
I am happy to explain a bit more if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Bump org.apache.solr:solr-core from 8.11.1 to 8.11.3 in /pulsar-io/solr [pulsar]
github-actions[bot] commented on PR #22047: URL: https://github.com/apache/pulsar/pull/22047#issuecomment-1936620319 @dependabot[bot] Please add the following content to your PR description and select a checkbox: ``` - [ ] `doc` - [ ] `doc-required` - [ ] `doc-not-needed` - [ ] `doc-complete` ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch dependabot/maven/pulsar-io/solr/org.apache.solr-solr-core-8.11.3 created (now 1929569fb6a)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/pulsar-io/solr/org.apache.solr-solr-core-8.11.3 in repository https://gitbox.apache.org/repos/asf/pulsar.git at 1929569fb6a Bump org.apache.solr:solr-core from 8.11.1 to 8.11.3 in /pulsar-io/solr No new revisions were added by this update.
[PR] Bump org.apache.solr:solr-core from 8.11.1 to 8.11.3 in /pulsar-io/solr [pulsar]
dependabot[bot] opened a new pull request, #22047: URL: https://github.com/apache/pulsar/pull/22047 Bumps org.apache.solr:solr-core from 8.11.1 to 8.11.3. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/pulsar/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [Bug] Reference docs is 404'd [pulsar]
withinboredom opened a new issue, #22046: URL: https://github.com/apache/pulsar/issues/22046 ### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Version https://pulsar.apache.org/docs/3.2.x/reference-configuration/#broker ### Minimal reproduce step Go to https://pulsar.apache.org/docs/3.2.x/reference-configuration/#broker and try to read the broker docs here: https://pulsar.apache.org/reference/#/config/reference-configuration-broker ### What did you expect to see? Some documentation helping me to configure things ### What did you see instead? Nothing useful ### Anything else? No ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker][branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter [pulsar]
frankjkelly commented on PR #22011: URL: https://github.com/apache/pulsar/pull/22011#issuecomment-1936543593 > > > > Thanks for the clarification so does that mean the client will retry and if so is that within milliseconds, or seconds or something else? > > > > > > > > > @frankjkelly I think the client does not retry automatically and the user needs to resend the message manually if message sent fails. > > > > > > Hmmm @merlimat or @lhotari can you confirm? If this error requires the caller to catch and retry (as opposed to the client doing it internally) then that's a concern for adoption of the rate limiter (if the error occurs and the client retries as best it can that's OK). > > I don't see anything special about rate limiters in message delivery and retries. The Pulsar client is designed to continue attempting to send messages until a potential send timeout occurs. It's also possible to set up an unlimited send timeout, allowing the client to retry indefinitely. This feature is detailed in the Pulsar documentation, available at https://pulsar.apache.org/docs/3.1.x/cookbooks-deduplication/#pulsar-clients (it's explained in the context of message deduplication). You can refer to the Javadocs for [sendTimeout on ProducerBuilder](https://pulsar.apache.org/api/client/3.1.x/org/apache/pulsar/client/api/ProducerBuilder.html#sendTimeout(int,java.util.concurrent.TimeUnit)). > > It's crucial for messaging applications to be equipped to handle potential failures in message delivery, especially when data consistency is a key concern. Once the Pulsar client has acknowledged the message as sent by returning the message id, the responsibility for maintaining and ensuring the delivery of the message shifts to Pulsar. It's also necessary to verify that the message id is returned when using the asynchronous API ([sendAsync](https://pulsar.apache.org/api/client/3.1.x/org/apache/pulsar/client/api/Producer.html#sendAsync(T))). > > If sending results in an error or the messaging application never receives a message id from the Pulsar client, it's the messaging application's responsibility to retry. > > @frankjkelly, did I answer your question? You did - Thanks @lhotari -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [feat][misc] PIP-320: Add OpenTelemetry scaffolding [pulsar]
codecov-commenter commented on PR #22010: URL: https://github.com/apache/pulsar/pull/22010#issuecomment-1936447675 ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22010?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report Attention: `9 lines` in your changes are missing coverage. Please review. > Comparison is base [(`a83e9ed`)](https://app.codecov.io/gh/apache/pulsar/commit/a83e9ed70323b80319545326716340ce76d690eb?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) 36.63% compared to head [(`7241867`)](https://app.codecov.io/gh/apache/pulsar/pull/22010?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) 73.61%. > Report is 2 commits behind head on master. Additional details and impacted files [](https://app.codecov.io/gh/apache/pulsar/pull/22010?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) ```diff @@ Coverage Diff @@ ## master #22010 +/- ## = + Coverage 36.63% 73.61% +36.98% - Complexity1244532547+20102 = Files 1729 1870 +141 Lines132076 139039 +6963 Branches 1445215245 +793 = + Hits 48381 102359+53978 + Misses7729628776-48520 - Partials 6399 7904 +1505 ``` | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/22010/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/22010/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `24.26% <64.78%> (+0.12%)` | :arrow_up: | | [systests](https://app.codecov.io/gh/apache/pulsar/pull/22010/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `23.98% <69.01%> (-0.03%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/22010/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `72.91% <87.32%> (+40.88%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/apache/pulsar/pull/22010?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [.../pulsar/opentelemetry/OpenTelemetryAttributes.java](https://app.codecov.io/gh/apache/pulsar/pull/22010?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLW9wZW50ZWxlbWV0cnkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9vcGVudGVsZW1ldHJ5L09wZW5UZWxlbWV0cnlBdHRyaWJ1dGVzLmphdmE=) | `100.00% <100.00%> (ø)` | | | [...n/java/org/apache/pulsar/broker/PulsarService.java](https://app.codecov.io/gh/apache/pulsar/pull/22010?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9QdWxzYXJTZXJ2aWNlLmphdmE=) | `82.04% <75.00%> (+13.08%)` | :arrow_up: | | [...pulsar/broker/stats/PulsarBrokerOpenTelemetry.java](https://app.codecov.io/gh/apache/pulsar/pull/22010?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zdGF0cy9QdWxzYXJCcm9rZXJPcGVuVGVsZW1ldHJ5LmphdmE=) | `90.90% <90.90%> (ø)` | | | [...ar/functions/worker/PulsarWorkerOpenTelemetry.java](https://app.codecov.io/gh/apache/pulsar/pull/22010?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLWZ1bmN0aW9ucy93b3JrZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9mdW5jdGlvbnMvd29ya2VyL1B1bHNhcldvcmtlck9wZW5UZWxlbWV0cnkuamF2YQ==) | `90.90% <90.90%> (ø)` | | | [...a/org/apache/pulsar/proxy/server/ProxyService.java](https://app.codecov.io/gh/apache/pulsar/pull/22010?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-cHVsc2FyLXByb3h5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvcHJveHkvc2VydmVyL1Byb3h5U2VydmljZS5qYXZh) | `79.16% <75.00%> (+29.16%)` |
Re: [PR] [improve] PIP-335: Pulsar with Oxia integration test [pulsar]
merlimat merged PR #22045: URL: https://github.com/apache/pulsar/pull/22045 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar) branch master updated: [improve] PIP-335: Pulsar with Oxia integration test (#22045)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 2b75ca0e02c [improve] PIP-335: Pulsar with Oxia integration test (#22045) 2b75ca0e02c is described below commit 2b75ca0e02c10262813de509b96f5678faffc934 Author: Matteo Merli AuthorDate: Fri Feb 9 10:50:33 2024 -0800 [improve] PIP-335: Pulsar with Oxia integration test (#22045) --- .../docker-images/latest-version-image/Dockerfile | 2 +- .../latest-version-image/scripts/init-cluster.sh | 38 --- .../latest-version-image/scripts/run-bookie.sh | 1 - .../latest-version-image/scripts/run-broker.sh | 1 - .../scripts/run-functions-worker.sh| 1 - .../latest-version-image/scripts/run-proxy.sh | 1 - .../latest-version-image/scripts/run-websocket.sh | 1 - .../containers/PulsarInitMetadataContainer.java| 76 + .../tests/integration/oxia/OxiaContainer.java | 72 + .../tests/integration/oxia/OxiaSmokeTest.java | 48 +++ .../integration/topologies/PulsarCluster.java | 330 - .../integration/topologies/PulsarClusterSpec.java | 3 + .../src/test/resources/pulsar-messaging.xml| 2 + 13 files changed, 397 insertions(+), 179 deletions(-) diff --git a/tests/docker-images/latest-version-image/Dockerfile b/tests/docker-images/latest-version-image/Dockerfile index 4973bec0441..f019af5c926 100644 --- a/tests/docker-images/latest-version-image/Dockerfile +++ b/tests/docker-images/latest-version-image/Dockerfile @@ -50,7 +50,7 @@ COPY conf/supervisord.conf /etc/supervisord.conf COPY conf/global-zk.conf conf/local-zk.conf conf/bookie.conf conf/broker.conf conf/functions_worker.conf \ conf/proxy.conf conf/websocket.conf /etc/supervisord/conf.d/ -COPY scripts/init-cluster.sh scripts/run-global-zk.sh scripts/run-local-zk.sh \ +COPY scripts/run-global-zk.sh scripts/run-local-zk.sh \ scripts/run-bookie.sh scripts/run-broker.sh scripts/run-functions-worker.sh scripts/run-proxy.sh \ scripts/run-standalone.sh scripts/run-websocket.sh \ /pulsar/bin/ diff --git a/tests/docker-images/latest-version-image/scripts/init-cluster.sh b/tests/docker-images/latest-version-image/scripts/init-cluster.sh deleted file mode 100755 index 926845d5a77..000 --- a/tests/docker-images/latest-version-image/scripts/init-cluster.sh +++ /dev/null @@ -1,38 +0,0 @@ -#!/usr/bin/env bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. -# - -set -x - -ZNODE="/initialized-$clusterName" - -bin/watch-znode.py -z $zkServers -p / -w - -bin/watch-znode.py -z $zkServers -p $ZNODE -e -if [ $? != 0 ]; then -echo Initializing cluster -bin/apply-config-from-env.py conf/bookkeeper.conf && -bin/pulsar initialize-cluster-metadata --cluster $clusterName --zookeeper $zkServers \ - --configuration-store $configurationStore --web-service-url http://$pulsarNode:8080/ \ - --broker-service-url pulsar://$pulsarNode:6650/ && -bin/watch-znode.py -z $zkServers -p $ZNODE -c -echo Initialized -else -echo Already Initialized -fi diff --git a/tests/docker-images/latest-version-image/scripts/run-bookie.sh b/tests/docker-images/latest-version-image/scripts/run-bookie.sh index 64466eb2d9a..e454e667645 100755 --- a/tests/docker-images/latest-version-image/scripts/run-bookie.sh +++ b/tests/docker-images/latest-version-image/scripts/run-bookie.sh @@ -29,6 +29,5 @@ if [ -z "$NO_AUTOSTART" ]; then sed -i 's/autostart=.*/autostart=true/' /etc/supervisord/conf.d/bookie.conf fi -bin/watch-znode.py -z $zkServers -p /initialized-$clusterName -w exec /usr/bin/supervisord -c /etc/supervisord.conf diff --git a/tests/docker-images/latest-version-image/scripts/run-broker.sh b/tests/docker-images/latest-version-image/scripts/run-broker.sh index 6ed5d60c39e..4f89f145f2b 100755 --- a/tests/docker-images/latest-version-image/scripts/run-broker.sh +++ b/tests/docker-images/latest-version-image/scripts/run-broker.sh @@ -25,6 +25,5 @@ if [ -z "$NO_AUTOSTART" ]; then sed -i
Re: [PR] [improve] PIP-335: Pulsar with Oxia integration test [pulsar]
codecov-commenter commented on PR #22045: URL: https://github.com/apache/pulsar/pull/22045#issuecomment-1936438602 ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22045?src=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Comparison is base [(`a83e9ed`)](https://app.codecov.io/gh/apache/pulsar/commit/a83e9ed70323b80319545326716340ce76d690eb?el=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) 36.63% compared to head [(`d59c952`)](https://app.codecov.io/gh/apache/pulsar/pull/22045?src=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) 73.69%. > Report is 1 commits behind head on master. Additional details and impacted files [](https://app.codecov.io/gh/apache/pulsar/pull/22045?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) ```diff @@ Coverage Diff @@ ## master #22045 +/- ## = + Coverage 36.63% 73.69% +37.06% - Complexity1244532442+19997 = Files 1729 1865 +136 Lines132076 138968 +6892 Branches 1445215236 +784 = + Hits 48381 102415+54034 + Misses7729628651-48645 - Partials 6399 7902 +1503 ``` | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/22045/flags?src=pr=flags_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | Coverage Δ | | |---|---|---| | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/22045/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `24.47% <ø> (+0.33%)` | :arrow_up: | | [systests](https://app.codecov.io/gh/apache/pulsar/pull/22045/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `24.30% <ø> (+0.29%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/22045/flags?src=pr=flag_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) | `72.96% <ø> (+40.93%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. [see 1442 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/22045/indirect-changes?src=pr=tree-more_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Delete current ledger when it is closed [pulsar]
lhotari commented on PR #22034: URL: https://github.com/apache/pulsar/pull/22034#issuecomment-1936364633 There's a scheduled job https://github.com/apache/pulsar/blob/5df97b41a431a714a448c198c312e98aa4a084e8/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L653-L659 that runs by default every 120 seconds (configured with `retentionCheckIntervalInSeconds`). That iterates all active topics and their managed ledgers: https://github.com/apache/pulsar/blob/5df97b41a431a714a448c198c312e98aa4a084e8/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L2041-L2051 This works as long as the topic is active (loaded) in a broker in the cluster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [feat][misc] PIP-320: Add OpenTelemetry scaffolding [pulsar]
dragosvictor commented on code in PR #22010:
URL: https://github.com/apache/pulsar/pull/22010#discussion_r1484638484
##
pulsar-opentelemetry/src/test/java/org/apache/pulsar/opentelemetry/OpenTelemetryServiceTest.java:
##
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.opentelemetry;
+
+import static
io.opentelemetry.sdk.testing.assertj.OpenTelemetryAssertions.assertThat;
+import static
io.opentelemetry.sdk.testing.assertj.OpenTelemetryAssertions.satisfies;
+import io.opentelemetry.api.common.AttributeKey;
+import io.opentelemetry.api.common.Attributes;
+import io.opentelemetry.api.metrics.LongCounterBuilder;
+import io.opentelemetry.api.metrics.Meter;
+import
io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder;
+import io.opentelemetry.sdk.common.InstrumentationScopeInfo;
+import io.opentelemetry.sdk.metrics.export.MetricReader;
+import io.opentelemetry.sdk.testing.exporter.InMemoryMetricReader;
+import io.opentelemetry.semconv.ResourceAttributes;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import lombok.Cleanup;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsClient;
+import org.assertj.core.api.AbstractCharSequenceAssert;
+import org.awaitility.Awaitility;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+public class OpenTelemetryServiceTest {
+
+private OpenTelemetryService openTelemetryService;
+private InMemoryMetricReader reader;
+private Meter meter;
+
+@BeforeMethod
+public void setup() throws Exception {
+reader = InMemoryMetricReader.create();
+openTelemetryService = OpenTelemetryService.builder().
+sdkBuilderConsumer(getSdkBuilderConsumer(reader,
+Map.of(OpenTelemetryService.OTEL_SDK_DISABLED_KEY,
"false"))).
+clusterName("openTelemetryServiceTestCluster").
+build();
+meter =
openTelemetryService.getOpenTelemetry().getMeter("openTelemetryServiceTestInstrument");
+}
+
+@AfterMethod
+public void teardown() throws Exception {
+openTelemetryService.close();
+reader.close();
+}
+
+// Customizes the SDK builder to include the MetricReader and extra
properties for testing purposes.
+private static Consumer
getSdkBuilderConsumer(MetricReader extraReader,
+
Map extraProperties) {
+return autoConfigurationCustomizer -> {
+if (extraReader != null) {
+autoConfigurationCustomizer.addMeterProviderCustomizer(
+(sdkMeterProviderBuilder, __) ->
sdkMeterProviderBuilder.registerMetricReader(extraReader));
+}
+autoConfigurationCustomizer.addPropertiesSupplier(() ->
extraProperties);
+};
+}
+
+@Test(expectedExceptions = IllegalArgumentException.class)
+public void testClusterNameCannotBeNull() {
+@Cleanup
+var ots = OpenTelemetryService.builder().build();
+}
+
+@Test(expectedExceptions = IllegalArgumentException.class)
+public void testClusterNameCannotBeEmpty() {
+@Cleanup
+var ots =
OpenTelemetryService.builder().clusterName(StringUtils.EMPTY).build();
+}
+
+@Test
+public void testResourceAttributesAreSet() throws Exception {
+@Cleanup
+var reader = InMemoryMetricReader.create();
+
+@Cleanup
+var ots = OpenTelemetryService.builder().
+sdkBuilderConsumer(getSdkBuilderConsumer(reader,
+Map.of(OpenTelemetryService.OTEL_SDK_DISABLED_KEY,
"false"))).
+clusterName("testServiceNameAndVersion").
+serviceName("openTelemetryServiceTestService").
+serviceVersion("1.0.0").
+build();
+
+assertThat(reader.collectAllMetrics())
+.allSatisfy(metric ->
Re: [PR] [fix][broker] Delete current ledger when it is closed [pulsar]
lhotari commented on PR #22034: URL: https://github.com/apache/pulsar/pull/22034#issuecomment-1936321236 It seems that the problem might be very different. It would be necessary to explain the use case. The description doesn't explain the use case. If I have understood correctly, the use case is about configuring retention by time. When retention is configured, it is expected that expired messages get deleted. That isn't happening in some cases currently and that's the issue that is being addressed. One rare case where Pulsar currently seems to ignore retention policies is the case when topics aren't actively loaded on any broker in the cluster. A topic gets loaded when a consumer or producer connects to the topic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker] Delete current ledger when it is closed [pulsar]
lhotari commented on PR #22034: URL: https://github.com/apache/pulsar/pull/22034#issuecomment-1936293921 Would it make sense to rename "Delete current ledger when it is closed" to "Trim current ledger when it is closed" if that's what is the expectation? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve] PIP-335: Pulsar with Oxia integration test [pulsar]
merlimat commented on code in PR #22045:
URL: https://github.com/apache/pulsar/pull/22045#discussion_r1484578470
##
tests/integration/src/test/java/org/apache/pulsar/tests/integration/containers/PulsarInitMetadataContainer.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.tests.integration.containers;
+
+import java.io.IOException;
+import lombok.extern.slf4j.Slf4j;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.Network;
+
+/**
+ * Initialize the Pulsar metadata
+ */
+@Slf4j
+public class PulsarInitMetadataContainer extends
GenericContainer {
+
+public static final String NAME = "init-metadata";
+
+private final String clusterName;
+private final String metadataStoreUrl;
+private final String configurationMetadataStoreUrl;
+private final String brokerHostname;
+
+public PulsarInitMetadataContainer(Network network,
+ String clusterName,
+ String metadataStoreUrl,
+ String configurationMetadataStoreUrl,
+ String brokerHostname) {
+this.clusterName = clusterName;
+this.metadataStoreUrl = metadataStoreUrl;
+this.configurationMetadataStoreUrl = configurationMetadataStoreUrl;
+this.brokerHostname = brokerHostname;
+setDockerImageName(PulsarContainer.DEFAULT_IMAGE_NAME);
+withNetwork(network);
+
+setCommand("sleep 100");
Review Comment:
Yes, this spins up a container just idle. The command to initialize the
cluster is then `execd` into the container, to retrieve the exit code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [improve] PIP-335: Pulsar with Oxia integration test [pulsar]
heesung-sn commented on code in PR #22045:
URL: https://github.com/apache/pulsar/pull/22045#discussion_r1484569644
##
tests/integration/src/test/java/org/apache/pulsar/tests/integration/containers/PulsarInitMetadataContainer.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.tests.integration.containers;
+
+import java.io.IOException;
+import lombok.extern.slf4j.Slf4j;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.Network;
+
+/**
+ * Initialize the Pulsar metadata
+ */
+@Slf4j
+public class PulsarInitMetadataContainer extends
GenericContainer {
+
+public static final String NAME = "init-metadata";
+
+private final String clusterName;
+private final String metadataStoreUrl;
+private final String configurationMetadataStoreUrl;
+private final String brokerHostname;
+
+public PulsarInitMetadataContainer(Network network,
+ String clusterName,
+ String metadataStoreUrl,
+ String configurationMetadataStoreUrl,
+ String brokerHostname) {
+this.clusterName = clusterName;
+this.metadataStoreUrl = metadataStoreUrl;
+this.configurationMetadataStoreUrl = configurationMetadataStoreUrl;
+this.brokerHostname = brokerHostname;
+setDockerImageName(PulsarContainer.DEFAULT_IMAGE_NAME);
+withNetwork(network);
+
+setCommand("sleep 100");
Review Comment:
nit: Do we need this sleep command?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] Improve the internal Jersey Client to use a Connection Pool [pulsar]
lhotari commented on issue #22003: URL: https://github.com/apache/pulsar/issues/22003#issuecomment-1936259756 This issue seems to be a duplicate of #22041. The AsyncHttpClient is used as a Jersey client connector in pulsar-client-admin. I'll close this ticket since #22041 contains a better issue description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Improve the internal Jersey Client to use a Connection Pool [pulsar]
lhotari closed issue #22003: Improve the internal Jersey Client to use a Connection Pool URL: https://github.com/apache/pulsar/issues/22003 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [improve] PIP-335: Pulsar with Oxia integration test [pulsar]
heesung-sn commented on code in PR #22045:
URL: https://github.com/apache/pulsar/pull/22045#discussion_r1484569644
##
tests/integration/src/test/java/org/apache/pulsar/tests/integration/containers/PulsarInitMetadataContainer.java:
##
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pulsar.tests.integration.containers;
+
+import java.io.IOException;
+import lombok.extern.slf4j.Slf4j;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.Network;
+
+/**
+ * Initialize the Pulsar metadata
+ */
+@Slf4j
+public class PulsarInitMetadataContainer extends
GenericContainer {
+
+public static final String NAME = "init-metadata";
+
+private final String clusterName;
+private final String metadataStoreUrl;
+private final String configurationMetadataStoreUrl;
+private final String brokerHostname;
+
+public PulsarInitMetadataContainer(Network network,
+ String clusterName,
+ String metadataStoreUrl,
+ String configurationMetadataStoreUrl,
+ String brokerHostname) {
+this.clusterName = clusterName;
+this.metadataStoreUrl = metadataStoreUrl;
+this.configurationMetadataStoreUrl = configurationMetadataStoreUrl;
+this.brokerHostname = brokerHostname;
+setDockerImageName(PulsarContainer.DEFAULT_IMAGE_NAME);
+withNetwork(network);
+
+setCommand("sleep 100");
Review Comment:
Do we need this sleep command?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] Improve the internal Jersey Client to use a Connection Pool [pulsar]
lhotari commented on issue #22003: URL: https://github.com/apache/pulsar/issues/22003#issuecomment-1936252205 pulsar-client-admin uses AsyncHttpClient , I created #22041 for addressing that. Jetty client is used in the Pulsar Proxy [for proxying /admin endpoint](https://github.com/apache/pulsar/blob/master/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/AdminProxyHandler.java). Where do we have a Jersey client? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix] [broker] [branch-3.0] Fast fix infinite HTTP call createSubscriptions caused by wrong topicName [pulsar]
lhotari commented on PR #21997: URL: https://github.com/apache/pulsar/pull/21997#issuecomment-1936239338 It was also noticed that pulsar-client-admin doesn't have http connection pooling configured resulting in "Connection Limit(2048) reached for " messages in the log. There is #22041 to add connection pooling to pulsar-client-admin so that fewer connections would be created to brokers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [improve] PIP-335: Pulsar with Oxia integration test [pulsar]
merlimat opened a new pull request, #22045: URL: https://github.com/apache/pulsar/pull/22045 PIP: [335](https://github.com/apache/pulsar/blob/master/pip/pip-335%20Oxia%20metadata%20plugin.md) ### Motivation Added integration test for Pulsar & BookKeeper cluster with Oxia metadata store. ### Modifications * Removed the `init-cluster.sh` script that was invoked in the ZK container. Instead use a separate container and call `pulsar initialize-cluster-metadata` from the Java code. ### Does this pull request potentially affect one of the following parts: *If the box was checked, please highlight the changes* - [ ] Dependencies (add or upgrade a dependency) - [ ] The public API - [ ] The schema - [ ] The default values of configurations - [ ] The threading model - [ ] The binary protocol - [ ] The REST endpoints - [ ] The admin CLI options - [ ] The metrics - [ ] Anything that affects deployment ### Documentation - [ ] `doc` - [ ] `doc-required` - [x] `doc-not-needed` - [ ] `doc-complete` ### Matching PR in forked repository PR in forked repository: https://github.com/merlimat/pulsar/pull/11 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][broker][branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter [pulsar]
lhotari commented on PR #22011: URL: https://github.com/apache/pulsar/pull/22011#issuecomment-1936188709 > > > Thanks for the clarification so does that mean the client will retry and if so is that within milliseconds, or seconds or something else? > > > > > > @frankjkelly I think the client does not retry automatically and the user needs to resend the message manually if message sent fails. > > Hmmm @merlimat or @lhotari can you confirm? If this error requires the caller to catch and retry (as opposed to the client doing it internally) then that's a concern for adoption of the rate limiter (if the error occurs and the client retries as best it can that's OK). I don't see anything special about rate limiters in message delivery and retries. The Pulsar client is designed to continue attempting to send messages until a potential send timeout occurs. It's also possible to set up an unlimited send timeout, allowing the client to retry indefinitely. This feature is detailed in the Pulsar documentation, available at https://pulsar.apache.org/docs/3.1.x/cookbooks-deduplication/#pulsar-clients (it's explained in the context of message deduplication). You can refer to the Javadocs for [sendTimeout on ProducerBuilder](https://pulsar.apache.org/api/client/3.1.x/org/apache/pulsar/client/api/ProducerBuilder.html#sendTimeout(int,java.util.concurrent.TimeUnit)). It's crucial for messaging applications to be equipped to handle potential failures in message delivery, especially when data consistency is a key concern. Once the Pulsar client has acknowledged the message as sent by returning the message id, the responsibility for maintaining and ensuring the delivery of the message shifts to Pulsar. It's also necessary to verify that the message id is returned when using the asynchronous API ([sendAsync](https://pulsar.apache.org/api/client/3.1.x/org/apache/pulsar/client/api/Producer.html#sendAsync(T))). If sending results in an error or the messaging application never receives a message id from the Pulsar client, it's the messaging application's responsibility to retry. @frankjkelly, did I answer your question? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Bug] Bookkeeper autorecovery ReplicationWorker not rereplicating under replicated ledgers. [pulsar]
horizonzy commented on issue #21987: URL: https://github.com/apache/pulsar/issues/21987#issuecomment-1936146542 Hi, Thomas. Sorry for the late reply. I check the recovery jvm stack, I found that the ReplicationWorker didn't find any underreplication ledger, it is waiting for the incoming underreplication ledger. And the metadata store executor didn't block, so if the Auditor find unnderreplication ledger, the ReplicationWorker should be notified, it will get the unnderreplication ledger to replicate. But from your listunderreplicated result, it show that there are 1885 underreplication ledger, so I guess that the ReplicationWorker using a wrong zk url, it didn't find the unnderreplication ledger. Could you help to check it, and give the heap dump file. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [fix][proxy] Fix outboundChannel close [pulsar]
dao-jun commented on PR #22015: URL: https://github.com/apache/pulsar/pull/22015#issuecomment-1936080318 seems netty's issue, but I've checked netty's source code, this is just a WARN log. but if we want to fix it, maybe we can change the [line](https://github.com/netty/netty/blob/4.1/transport-classes-epoll/src/main/java/io/netty/channel/epoll/AbstractEpollStreamChannel.java#L894) to: `promise.tryFailure(cause)` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(pulsar-site) branch main updated: Fix missing docker cmd (#785)
This is an automated email from the ASF dual-hosted git repository. technoboy pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git The following commit(s) were added to refs/heads/main by this push: new 44a3cd53b927 Fix missing docker cmd (#785) 44a3cd53b927 is described below commit 44a3cd53b92714c2e28d08121c95565e19e9a7bb Author: Jiwei Guo AuthorDate: Fri Feb 9 22:15:15 2024 +0800 Fix missing docker cmd (#785) --- docs/getting-started-docker.md | 1 + versioned_docs/version-3.2.x/getting-started-docker.md | 1 + 2 files changed, 2 insertions(+) diff --git a/docs/getting-started-docker.md b/docs/getting-started-docker.md index 7daa1b604e38..994b41f89c0f 100644 --- a/docs/getting-started-docker.md +++ b/docs/getting-started-docker.md @@ -28,6 +28,7 @@ For macOS, Linux, and Windows, run the following command to start Pulsar within ```shell +docker run -it \ -p 6650:6650 \ -p 8080:8080 \ --mount source=pulsardata,target=/pulsar/data \ diff --git a/versioned_docs/version-3.2.x/getting-started-docker.md b/versioned_docs/version-3.2.x/getting-started-docker.md index 7daa1b604e38..994b41f89c0f 100644 --- a/versioned_docs/version-3.2.x/getting-started-docker.md +++ b/versioned_docs/version-3.2.x/getting-started-docker.md @@ -28,6 +28,7 @@ For macOS, Linux, and Windows, run the following command to start Pulsar within ```shell +docker run -it \ -p 6650:6650 \ -p 8080:8080 \ --mount source=pulsardata,target=/pulsar/data \
Re: [I] [Doc] The pulsar docker deployment command is incorrect [pulsar]
Technoboy- closed issue #22027: [Doc] The pulsar docker deployment command is incorrect URL: https://github.com/apache/pulsar/issues/22027 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix missing docker cmd [pulsar-site]
Technoboy- merged PR #785: URL: https://github.com/apache/pulsar-site/pull/785 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [feat][io] New component. AzureDataExplorer/Kusto sink for pulsar (https://learn.microsoft.com/en-us/azure/data-explorer/) [pulsar]
asaharn commented on code in PR #22006:
URL: https://github.com/apache/pulsar/pull/22006#discussion_r1484293622
##
pulsar-io/azure-data-explorer/src/main/java/org/apache/pulsar/io/azuredataexplorer/ADXSinkConfig.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.io.azuredataexplorer;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import java.util.Objects;
+import lombok.Data;
+import lombok.experimental.Accessors;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+
+@Data
+@Accessors(chain = true)
+public class ADXSinkConfig implements Serializable {
+
+@FieldDoc(required = true, defaultValue = "", help = "The ADX cluster URL")
+private String clusterUrl;
+
+@FieldDoc(required = true, defaultValue = "", help = "The database name to
which data need to be ingested")
+private String database;
+
+@FieldDoc(required = true, defaultValue = "", help = "Table name to which
pulsar data need to be ingested.")
+private String table;
+
+@FieldDoc(defaultValue = "", help = "The AAD app Id for authentication")
+private String appId;
+
+@FieldDoc(defaultValue = "", help = "The AAD app secret for
authentication")
Review Comment:
Sorry @lhotari I missed this, will take care.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [feat][io] New component. AzureDataExplorer/Kusto sink for pulsar (https://learn.microsoft.com/en-us/azure/data-explorer/) [pulsar]
asaharn commented on code in PR #22006:
URL: https://github.com/apache/pulsar/pull/22006#discussion_r1484292510
##
pulsar-io/azure-data-explorer/src/main/java/org/apache/pulsar/io/azuredataexplorer/ADXSinkConfig.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.io.azuredataexplorer;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import java.util.Objects;
+import lombok.Data;
+import lombok.experimental.Accessors;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+
+@Data
+@Accessors(chain = true)
+public class ADXSinkConfig implements Serializable {
+
+@FieldDoc(required = true, defaultValue = "", help = "The ADX cluster URL")
+private String clusterUrl;
+
+@FieldDoc(required = true, defaultValue = "", help = "The database name to
which data need to be ingested")
+private String database;
+
+@FieldDoc(required = true, defaultValue = "", help = "Table name to which
pulsar data need to be ingested.")
+private String table;
+
+@FieldDoc(defaultValue = "", help = "The AAD app Id for authentication")
+private String appId;
+
+@FieldDoc(defaultValue = "", help = "The AAD app secret for
authentication")
+private String appKey;
+
+@FieldDoc(defaultValue = "", help = "The tenant Id for authentication")
+private String tenantId;
+
+@FieldDoc(defaultValue = "", help = "The Managed Identity credential for
authentication."
++ " Set this with clientId in case of User assigned MI."
++ " and 'system' in case of System assigned managed identity")
+private String managedIdentityId;
+
+@FieldDoc(defaultValue = "", help = "The mapping reference for ingestion")
+private String mappingRefName;
+
+@FieldDoc(defaultValue = "CSV", help = "The type of mapping reference
provided")
+private String mappingRefType;
+
+@FieldDoc(defaultValue = "false", help = "Denotes if flush should happen
immediately without aggregation. "
++ "Not recommended to enable flushImmediately for production
workloads")
+private boolean flushImmediately = false;
+
+@FieldDoc(defaultValue = "100", help = "For batching, this defines the
number of "
++ "records to hold for batching, to sink data to adx")
+private int batchSize = 100;
+
+@FieldDoc(defaultValue = "1", help = "For batching, this defines the
time to hold"
++ " records before sink to adx")
+private long batchTimeMs = 1;
+
+@FieldDoc(defaultValue = "1", help = "Max retry attempts, In case of
transient ingestion errors")
+private int maxRetryAttempts = 1;
+
+@FieldDoc(defaultValue = "10", help = "Period of time in milliseconds to
backoff"
++ " before retry for transient errors")
+private long retryBackOffTime = 10;
+
+
+public static ADXSinkConfig load(String yamlFile) throws IOException {
+ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
+return mapper.readValue(new File(yamlFile), ADXSinkConfig.class);
+}
+
+protected static ADXSinkConfig load(Map config) throws
JsonProcessingException {
+ObjectMapper mapper = new ObjectMapper();
+return mapper.readValue(mapper.writeValueAsString(config),
ADXSinkConfig.class);
+}
Review Comment:
Thanks for the explanation @lhotari , I checked the sources you provided.
Will implement the `IOConfigUtils.loadWithSecrets` way to load the sensitive
info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [fix][broker][branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter [pulsar]
frankjkelly commented on PR #22011: URL: https://github.com/apache/pulsar/pull/22011#issuecomment-1935889856 > > Thanks for the clarification so does that mean the client will retry and if so is that within milliseconds, or seconds or something else? > > @frankjkelly I think the client does not retry automatically and the user needs to resend the message manually if message sent fails. Hmmm @merlimat or @lhotari can you confirm? If this error requires the caller to catch and retry (as opposed to the client doing it internally) then that's a concern for adoption of the rate limiter (if the error occurs and the client retries as best it can that's OK). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [feat][io] New component. AzureDataExplorer/Kusto sink for pulsar (https://learn.microsoft.com/en-us/azure/data-explorer/) [pulsar]
lhotari commented on code in PR #22006:
URL: https://github.com/apache/pulsar/pull/22006#discussion_r1484216970
##
pulsar-io/azure-data-explorer/src/main/java/org/apache/pulsar/io/azuredataexplorer/ADXSinkConfig.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.io.azuredataexplorer;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import java.util.Objects;
+import lombok.Data;
+import lombok.experimental.Accessors;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+
+@Data
+@Accessors(chain = true)
+public class ADXSinkConfig implements Serializable {
+
+@FieldDoc(required = true, defaultValue = "", help = "The ADX cluster URL")
+private String clusterUrl;
+
+@FieldDoc(required = true, defaultValue = "", help = "The database name to
which data need to be ingested")
+private String database;
+
+@FieldDoc(required = true, defaultValue = "", help = "Table name to which
pulsar data need to be ingested.")
+private String table;
+
+@FieldDoc(defaultValue = "", help = "The AAD app Id for authentication")
+private String appId;
+
+@FieldDoc(defaultValue = "", help = "The AAD app secret for
authentication")
+private String appKey;
+
+@FieldDoc(defaultValue = "", help = "The tenant Id for authentication")
+private String tenantId;
+
+@FieldDoc(defaultValue = "", help = "The Managed Identity credential for
authentication."
++ " Set this with clientId in case of User assigned MI."
++ " and 'system' in case of System assigned managed identity")
+private String managedIdentityId;
+
+@FieldDoc(defaultValue = "", help = "The mapping reference for ingestion")
+private String mappingRefName;
+
+@FieldDoc(defaultValue = "CSV", help = "The type of mapping reference
provided")
+private String mappingRefType;
+
+@FieldDoc(defaultValue = "false", help = "Denotes if flush should happen
immediately without aggregation. "
++ "Not recommended to enable flushImmediately for production
workloads")
+private boolean flushImmediately = false;
+
+@FieldDoc(defaultValue = "100", help = "For batching, this defines the
number of "
++ "records to hold for batching, to sink data to adx")
+private int batchSize = 100;
+
+@FieldDoc(defaultValue = "1", help = "For batching, this defines the
time to hold"
++ " records before sink to adx")
+private long batchTimeMs = 1;
+
+@FieldDoc(defaultValue = "1", help = "Max retry attempts, In case of
transient ingestion errors")
+private int maxRetryAttempts = 1;
+
+@FieldDoc(defaultValue = "10", help = "Period of time in milliseconds to
backoff"
++ " before retry for transient errors")
+private long retryBackOffTime = 10;
+
+
+public static ADXSinkConfig load(String yamlFile) throws IOException {
+ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
+return mapper.readValue(new File(yamlFile), ADXSinkConfig.class);
+}
+
+protected static ADXSinkConfig load(Map config) throws
JsonProcessingException {
+ObjectMapper mapper = new ObjectMapper();
+return mapper.readValue(mapper.writeValueAsString(config),
ADXSinkConfig.class);
+}
Review Comment:
An alternative is to use
[PIP-289](https://github.com/apache/pulsar/blob/master/pip/pip-289.md) support
which wouldn't require code changes. However, that's primarily a solution that
can be used for existing connectors that don't have proper handling for
secrets. The doc is useful to read to get an overview of secrets handling
challenges in Pulsar connectors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact
Re: [PR] [feat][io] New component. AzureDataExplorer/Kusto sink for pulsar (https://learn.microsoft.com/en-us/azure/data-explorer/) [pulsar]
lhotari commented on code in PR #22006:
URL: https://github.com/apache/pulsar/pull/22006#discussion_r1484208319
##
pulsar-io/azure-data-explorer/src/main/java/org/apache/pulsar/io/azuredataexplorer/ADXSinkConfig.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.io.azuredataexplorer;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
+import java.io.File;
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Map;
+import java.util.Objects;
+import lombok.Data;
+import lombok.experimental.Accessors;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
+
+@Data
+@Accessors(chain = true)
+public class ADXSinkConfig implements Serializable {
+
+@FieldDoc(required = true, defaultValue = "", help = "The ADX cluster URL")
+private String clusterUrl;
+
+@FieldDoc(required = true, defaultValue = "", help = "The database name to
which data need to be ingested")
+private String database;
+
+@FieldDoc(required = true, defaultValue = "", help = "Table name to which
pulsar data need to be ingested.")
+private String table;
+
+@FieldDoc(defaultValue = "", help = "The AAD app Id for authentication")
+private String appId;
+
+@FieldDoc(defaultValue = "", help = "The AAD app secret for
authentication")
+private String appKey;
+
+@FieldDoc(defaultValue = "", help = "The tenant Id for authentication")
+private String tenantId;
+
+@FieldDoc(defaultValue = "", help = "The Managed Identity credential for
authentication."
++ " Set this with clientId in case of User assigned MI."
++ " and 'system' in case of System assigned managed identity")
+private String managedIdentityId;
+
+@FieldDoc(defaultValue = "", help = "The mapping reference for ingestion")
+private String mappingRefName;
+
+@FieldDoc(defaultValue = "CSV", help = "The type of mapping reference
provided")
+private String mappingRefType;
+
+@FieldDoc(defaultValue = "false", help = "Denotes if flush should happen
immediately without aggregation. "
++ "Not recommended to enable flushImmediately for production
workloads")
+private boolean flushImmediately = false;
+
+@FieldDoc(defaultValue = "100", help = "For batching, this defines the
number of "
++ "records to hold for batching, to sink data to adx")
+private int batchSize = 100;
+
+@FieldDoc(defaultValue = "1", help = "For batching, this defines the
time to hold"
++ " records before sink to adx")
+private long batchTimeMs = 1;
+
+@FieldDoc(defaultValue = "1", help = "Max retry attempts, In case of
transient ingestion errors")
+private int maxRetryAttempts = 1;
+
+@FieldDoc(defaultValue = "10", help = "Period of time in milliseconds to
backoff"
++ " before retry for transient errors")
+private long retryBackOffTime = 10;
+
+
+public static ADXSinkConfig load(String yamlFile) throws IOException {
+ObjectMapper mapper = new ObjectMapper(new YAMLFactory());
+return mapper.readValue(new File(yamlFile), ADXSinkConfig.class);
+}
+
+protected static ADXSinkConfig load(Map config) throws
JsonProcessingException {
+ObjectMapper mapper = new ObjectMapper();
+return mapper.readValue(mapper.writeValueAsString(config),
ADXSinkConfig.class);
+}
Review Comment:
Please check https://github.com/apache/pulsar/pull/21675 to see how
sensitive values are handled there. Another possibility is to use what you have
and after this retrieve the secret using the getSecret method on the
SinkContext.
##
pulsar-io/azure-data-explorer/src/main/java/org/apache/pulsar/io/azuredataexplorer/ADXSinkConfig.java:
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the
Re: [PR] [feat][misc] PIP-320: Add OpenTelemetry scaffolding [pulsar]
lhotari commented on code in PR #22010:
URL: https://github.com/apache/pulsar/pull/22010#discussion_r1484124406
##
pulsar-opentelemetry/src/test/java/org/apache/pulsar/opentelemetry/OpenTelemetryServiceTest.java:
##
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.opentelemetry;
+
+import static
io.opentelemetry.sdk.testing.assertj.OpenTelemetryAssertions.assertThat;
+import static
io.opentelemetry.sdk.testing.assertj.OpenTelemetryAssertions.satisfies;
+import io.opentelemetry.api.common.AttributeKey;
+import io.opentelemetry.api.common.Attributes;
+import io.opentelemetry.api.metrics.LongCounterBuilder;
+import io.opentelemetry.api.metrics.Meter;
+import
io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder;
+import io.opentelemetry.sdk.common.InstrumentationScopeInfo;
+import io.opentelemetry.sdk.metrics.export.MetricReader;
+import io.opentelemetry.sdk.testing.exporter.InMemoryMetricReader;
+import io.opentelemetry.semconv.ResourceAttributes;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import lombok.Cleanup;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsClient;
+import org.assertj.core.api.AbstractCharSequenceAssert;
+import org.awaitility.Awaitility;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+public class OpenTelemetryServiceTest {
+
+private OpenTelemetryService openTelemetryService;
+private InMemoryMetricReader reader;
+private Meter meter;
+
+@BeforeMethod
+public void setup() throws Exception {
+reader = InMemoryMetricReader.create();
+openTelemetryService = OpenTelemetryService.builder().
+sdkBuilderConsumer(getSdkBuilderConsumer(reader,
+Map.of(OpenTelemetryService.OTEL_SDK_DISABLED_KEY,
"false"))).
+clusterName("openTelemetryServiceTestCluster").
+build();
+meter =
openTelemetryService.getOpenTelemetry().getMeter("openTelemetryServiceTestInstrument");
+}
+
+@AfterMethod
+public void teardown() throws Exception {
+openTelemetryService.close();
+reader.close();
+}
+
+// Customizes the SDK builder to include the MetricReader and extra
properties for testing purposes.
+private static Consumer
getSdkBuilderConsumer(MetricReader extraReader,
+
Map extraProperties) {
+return autoConfigurationCustomizer -> {
+if (extraReader != null) {
+autoConfigurationCustomizer.addMeterProviderCustomizer(
+(sdkMeterProviderBuilder, __) ->
sdkMeterProviderBuilder.registerMetricReader(extraReader));
+}
+autoConfigurationCustomizer.addPropertiesSupplier(() ->
extraProperties);
+};
+}
+
+@Test(expectedExceptions = IllegalArgumentException.class)
+public void testClusterNameCannotBeNull() {
+@Cleanup
+var ots = OpenTelemetryService.builder().build();
+}
+
+@Test(expectedExceptions = IllegalArgumentException.class)
+public void testClusterNameCannotBeEmpty() {
+@Cleanup
+var ots =
OpenTelemetryService.builder().clusterName(StringUtils.EMPTY).build();
+}
+
+@Test
+public void testResourceAttributesAreSet() throws Exception {
+@Cleanup
+var reader = InMemoryMetricReader.create();
+
+@Cleanup
+var ots = OpenTelemetryService.builder().
+sdkBuilderConsumer(getSdkBuilderConsumer(reader,
+Map.of(OpenTelemetryService.OTEL_SDK_DISABLED_KEY,
"false"))).
+clusterName("testServiceNameAndVersion").
+serviceName("openTelemetryServiceTestService").
+serviceVersion("1.0.0").
+build();
+
+assertThat(reader.collectAllMetrics())
+.allSatisfy(metric ->
Re: [PR] [fix][broker][branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter [pulsar]
coderzc commented on PR #22011: URL: https://github.com/apache/pulsar/pull/22011#issuecomment-1935656478 > Thanks for the clarification so does that mean the client will retry and if so is that within milliseconds, or seconds or something else? @frankjkelly I think the client does not retry automatically and the user needs to resend the message manually if message sent fails. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [feat][misc] PIP-320: Add OpenTelemetry scaffolding [pulsar]
lhotari commented on code in PR #22010:
URL: https://github.com/apache/pulsar/pull/22010#discussion_r1484076947
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java:
##
@@ -897,6 +902,7 @@ public void start() throws PulsarServerException {
}
this.metricsGenerator = new MetricsGenerator(this);
+this.openTelemetry = new PulsarBrokerOpenTelemetry(config);
Review Comment:
We have have this discussion later. I'll resolve the comment so that we can
get the PR merged asap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [feat][misc] PIP-320: Add OpenTelemetry scaffolding [pulsar]
lhotari commented on code in PR #22010:
URL: https://github.com/apache/pulsar/pull/22010#discussion_r1484071300
##
pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java:
##
@@ -897,6 +902,7 @@ public void start() throws PulsarServerException {
}
this.metricsGenerator = new MetricsGenerator(this);
+this.openTelemetry = new PulsarBrokerOpenTelemetry(config);
Review Comment:
> If disabled, as it is by default, all the meters and instruments created
are no-op and backed by the same objects
How does this no-op creation happen and where could I learn more details
about this? Do you have references to documentation etc.?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [feat][misc] PIP-320: Add OpenTelemetry scaffolding [pulsar]
dragosvictor commented on code in PR #22010:
URL: https://github.com/apache/pulsar/pull/22010#discussion_r1484067271
##
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/stats/PulsarWorkerOpenTelemetry.java:
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.functions.instance.stats;
+
+import io.opentelemetry.api.metrics.Meter;
+import java.io.Closeable;
+import lombok.Getter;
+import org.apache.pulsar.common.stats.OpenTelemetryService;
+import org.apache.pulsar.functions.instance.InstanceConfig;
+
+public class PulsarWorkerOpenTelemetry implements Closeable {
+
+private final OpenTelemetryService openTelemetryService;
+
+@Getter
+private final Meter meter;
+
+public PulsarWorkerOpenTelemetry(InstanceConfig instanceConfig) {
+openTelemetryService =
OpenTelemetryService.builder().clusterName(instanceConfig.getClusterName()).build();
Review Comment:
Added instrumented resource providers
([commit](https://github.com/apache/pulsar/pull/22010/commits/7bfda76b1da83d6423b409dfc7cab20c59c76c76))
as described
[here](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/resources/library).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(pulsar) branch branch-2.11 updated: [fix][broker] Sanitize values before logging in apply-config-from-env.py script (#22044)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-2.11
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.11 by this push:
new 4d320e780b7 [fix][broker] Sanitize values before logging in
apply-config-from-env.py script (#22044)
4d320e780b7 is described below
commit 4d320e780b7c662dcf3e4eff44522b5711d10b34
Author: Lari Hotari
AuthorDate: Thu Feb 8 21:43:26 2024 -0800
[fix][broker] Sanitize values before logging in apply-config-from-env.py
script (#22044)
(cherry picked from commit 303678364eab538c16041214cae1588a5b2111d9)
---
.../scripts/apply-config-from-env-with-prefix.py | 103 -
docker/pulsar/scripts/apply-config-from-env.py | 57 ++--
2 files changed, 45 insertions(+), 115 deletions(-)
diff --git a/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
b/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
index 0890f1b8d7b..9943b283a9f 100755
--- a/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
+++ b/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
@@ -18,89 +18,22 @@
# under the License.
#
-##
-## Edit a properties config file and replace values based on
-## the ENV variables
-## export prefix_my-key=new-value
-## ./apply-config-from-env-with-prefix prefix_ file.conf
-##
-
-import os, sys
-
-if len(sys.argv) < 3:
-print('Usage: %s' % (sys.argv[0]))
-sys.exit(1)
-
-# Always apply env config to env scripts as well
-prefix = sys.argv[1]
-conf_files = sys.argv[2:]
-
-PF_ENV_DEBUG = (os.environ.get('PF_ENV_DEBUG','0') == '1')
-
-for conf_filename in conf_files:
-lines = [] # List of config file lines
-keys = {} # Map a key to its line number in the file
-
-# Load conf file
-for line in open(conf_filename):
-lines.append(line)
-line = line.strip()
-if not line or line.startswith('#'):
-continue
-
-try:
-k,v = line.split('=', 1)
-keys[k] = len(lines) - 1
-except:
-if PF_ENV_DEBUG:
-print("[%s] skip Processing %s" % (conf_filename, line))
-
-# Update values from Env
-for k in sorted(os.environ.keys()):
-v = os.environ[k].strip()
-
-# Hide the value in logs if is password.
-if "password" in k.lower():
-displayValue = ""
-else:
-displayValue = v
-
-if k.startswith(prefix):
-k = k[len(prefix):]
-if k in keys:
-print('[%s] Applying config %s = %s' % (conf_filename, k,
displayValue))
-idx = keys[k]
-lines[idx] = '%s=%s\n' % (k, v)
-
-
-# Ensure we have a new-line at the end of the file, to avoid issue
-# when appending more lines to the config
-lines.append('\n')
-
-# Add new keys from Env
-for k in sorted(os.environ.keys()):
-v = os.environ[k]
-if not k.startswith(prefix):
-continue
-
-# Hide the value in logs if is password.
-if "password" in k.lower():
-displayValue = ""
-else:
-displayValue = v
-
-k = k[len(prefix):]
-if k not in keys:
-print('[%s] Adding config %s = %s' % (conf_filename, k,
displayValue))
-lines.append('%s=%s\n' % (k, v))
-else:
-print('[%s] Updating config %s = %s' % (conf_filename, k,
displayValue))
-lines[keys[k]] = '%s=%s\n' % (k, v)
-
+
+# Edit a properties config file and replace values based on
+# the ENV variables
+# export prefix_my-key=new-value
+# ./apply-config-from-env-with-prefix prefix_ file.conf
+#
+# Environment variables that are prefixed with the command
+# line prefix will be used to updated file properties if
+# they exist and create new ones if they don't.
+#
+# Environment variables not prefixed will be used only to
+# update if they exist and ignored if they don't.
+
-# Store back the updated config in the same file
-f = open(conf_filename, 'w')
-for line in lines:
-f.write(line)
-f.close()
+# DEPRECATED: Use "apply-config-from-env.py --prefix MY_PREFIX_ conf_file"
instead
+# this is not a python script, but a bash script. Call
apply-config-from-env.py with the prefix argument
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
+"${SCRIPT_DIR}/apply-config-from-env.py" --prefix "$1" "${@:2}"
diff --git a/docker/pulsar/scripts/apply-config-from-env.py
b/docker/pulsar/scripts/apply-config-from-env.py
index b8b479fc15b..da51f05f8be 100755
---
(pulsar) branch branch-3.1 updated: [fix][broker] Sanitize values before logging in apply-config-from-env.py script (#22044)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 9a8761e204f [fix][broker] Sanitize values before logging in
apply-config-from-env.py script (#22044)
9a8761e204f is described below
commit 9a8761e204f90a6763ea24a460059c11725494ac
Author: Lari Hotari
AuthorDate: Thu Feb 8 21:43:26 2024 -0800
[fix][broker] Sanitize values before logging in apply-config-from-env.py
script (#22044)
(cherry picked from commit 303678364eab538c16041214cae1588a5b2111d9)
---
.../scripts/apply-config-from-env-with-prefix.py | 85 ++
docker/pulsar/scripts/apply-config-from-env.py | 57 +++
2 files changed, 32 insertions(+), 110 deletions(-)
diff --git a/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
b/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
index 58f6c989750..9943b283a9f 100755
--- a/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
+++ b/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
@@ -32,83 +32,8 @@
# update if they exist and ignored if they don't.
-import os
-import sys
-
-if len(sys.argv) < 3:
-print('Usage: %s [...]' % (sys.argv[0]))
-sys.exit(1)
-
-# Always apply env config to env scripts as well
-prefix = sys.argv[1]
-conf_files = sys.argv[2:]
-
-PF_ENV_DEBUG = (os.environ.get('PF_ENV_DEBUG','0') == '1')
-
-for conf_filename in conf_files:
-lines = [] # List of config file lines
-keys = {} # Map a key to its line number in the file
-
-# Load conf file
-for line in open(conf_filename):
-lines.append(line)
-line = line.strip()
-if not line or line.startswith('#'):
-continue
-
-try:
-k,v = line.split('=', 1)
-keys[k] = len(lines) - 1
-except:
-if PF_ENV_DEBUG:
-print("[%s] skip Processing %s" % (conf_filename, line))
-
-# Update values from Env
-for k in sorted(os.environ.keys()):
-v = os.environ[k].strip()
-
-# Hide the value in logs if is password.
-if "password" in k.lower():
-displayValue = ""
-else:
-displayValue = v
-
-if k.startswith(prefix):
-k = k[len(prefix):]
-if k in keys:
-print('[%s] Applying config %s = %s' % (conf_filename, k,
displayValue))
-idx = keys[k]
-lines[idx] = '%s=%s\n' % (k, v)
-
-
-# Ensure we have a new-line at the end of the file, to avoid issue
-# when appending more lines to the config
-lines.append('\n')
-
-# Add new keys from Env
-for k in sorted(os.environ.keys()):
-v = os.environ[k]
-if not k.startswith(prefix):
-continue
-
-# Hide the value in logs if is password.
-if "password" in k.lower():
-displayValue = ""
-else:
-displayValue = v
-
-k = k[len(prefix):]
-if k not in keys:
-print('[%s] Adding config %s = %s' % (conf_filename, k,
displayValue))
-lines.append('%s=%s\n' % (k, v))
-else:
-print('[%s] Updating config %s = %s' % (conf_filename, k,
displayValue))
-lines[keys[k]] = '%s=%s\n' % (k, v)
-
-
-# Store back the updated config in the same file
-f = open(conf_filename, 'w')
-for line in lines:
-f.write(line)
-f.close()
+# DEPRECATED: Use "apply-config-from-env.py --prefix MY_PREFIX_ conf_file"
instead
+# this is not a python script, but a bash script. Call
apply-config-from-env.py with the prefix argument
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
+"${SCRIPT_DIR}/apply-config-from-env.py" --prefix "$1" "${@:2}"
diff --git a/docker/pulsar/scripts/apply-config-from-env.py
b/docker/pulsar/scripts/apply-config-from-env.py
index b8b479fc15b..da51f05f8be 100755
--- a/docker/pulsar/scripts/apply-config-from-env.py
+++ b/docker/pulsar/scripts/apply-config-from-env.py
@@ -25,18 +25,29 @@
## ./apply-config-from-env file.conf
##
-import os, sys
-
-if len(sys.argv) < 2:
-print('Usage: %s' % (sys.argv[0]))
+import os, sys, argparse
+
+parser = argparse.ArgumentParser(description='Pulsar configuration file
customizer based on environment variables')
+parser.add_argument('--prefix', default='PULSAR_PREFIX_', help='Prefix for
environment variables, default is PULSAR_PREFIX_')
+parser.add_argument('conf_files', nargs='*', help='Configuration files')
+args = parser.parse_args()
+if not args.conf_files:
+parser.print_help()
(pulsar) branch branch-3.2 updated: [fix][broker] Sanitize values before logging in apply-config-from-env.py script (#22044)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 14342058803 [fix][broker] Sanitize values before logging in
apply-config-from-env.py script (#22044)
14342058803 is described below
commit 143420588038c9468b2d9bad232c85c8ae05d003
Author: Lari Hotari
AuthorDate: Thu Feb 8 21:43:26 2024 -0800
[fix][broker] Sanitize values before logging in apply-config-from-env.py
script (#22044)
(cherry picked from commit 303678364eab538c16041214cae1588a5b2111d9)
---
.../scripts/apply-config-from-env-with-prefix.py | 85 ++
docker/pulsar/scripts/apply-config-from-env.py | 57 +++
2 files changed, 32 insertions(+), 110 deletions(-)
diff --git a/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
b/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
index 58f6c989750..9943b283a9f 100755
--- a/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
+++ b/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
@@ -32,83 +32,8 @@
# update if they exist and ignored if they don't.
-import os
-import sys
-
-if len(sys.argv) < 3:
-print('Usage: %s [...]' % (sys.argv[0]))
-sys.exit(1)
-
-# Always apply env config to env scripts as well
-prefix = sys.argv[1]
-conf_files = sys.argv[2:]
-
-PF_ENV_DEBUG = (os.environ.get('PF_ENV_DEBUG','0') == '1')
-
-for conf_filename in conf_files:
-lines = [] # List of config file lines
-keys = {} # Map a key to its line number in the file
-
-# Load conf file
-for line in open(conf_filename):
-lines.append(line)
-line = line.strip()
-if not line or line.startswith('#'):
-continue
-
-try:
-k,v = line.split('=', 1)
-keys[k] = len(lines) - 1
-except:
-if PF_ENV_DEBUG:
-print("[%s] skip Processing %s" % (conf_filename, line))
-
-# Update values from Env
-for k in sorted(os.environ.keys()):
-v = os.environ[k].strip()
-
-# Hide the value in logs if is password.
-if "password" in k.lower():
-displayValue = ""
-else:
-displayValue = v
-
-if k.startswith(prefix):
-k = k[len(prefix):]
-if k in keys:
-print('[%s] Applying config %s = %s' % (conf_filename, k,
displayValue))
-idx = keys[k]
-lines[idx] = '%s=%s\n' % (k, v)
-
-
-# Ensure we have a new-line at the end of the file, to avoid issue
-# when appending more lines to the config
-lines.append('\n')
-
-# Add new keys from Env
-for k in sorted(os.environ.keys()):
-v = os.environ[k]
-if not k.startswith(prefix):
-continue
-
-# Hide the value in logs if is password.
-if "password" in k.lower():
-displayValue = ""
-else:
-displayValue = v
-
-k = k[len(prefix):]
-if k not in keys:
-print('[%s] Adding config %s = %s' % (conf_filename, k,
displayValue))
-lines.append('%s=%s\n' % (k, v))
-else:
-print('[%s] Updating config %s = %s' % (conf_filename, k,
displayValue))
-lines[keys[k]] = '%s=%s\n' % (k, v)
-
-
-# Store back the updated config in the same file
-f = open(conf_filename, 'w')
-for line in lines:
-f.write(line)
-f.close()
+# DEPRECATED: Use "apply-config-from-env.py --prefix MY_PREFIX_ conf_file"
instead
+# this is not a python script, but a bash script. Call
apply-config-from-env.py with the prefix argument
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
+"${SCRIPT_DIR}/apply-config-from-env.py" --prefix "$1" "${@:2}"
diff --git a/docker/pulsar/scripts/apply-config-from-env.py
b/docker/pulsar/scripts/apply-config-from-env.py
index b8b479fc15b..da51f05f8be 100755
--- a/docker/pulsar/scripts/apply-config-from-env.py
+++ b/docker/pulsar/scripts/apply-config-from-env.py
@@ -25,18 +25,29 @@
## ./apply-config-from-env file.conf
##
-import os, sys
-
-if len(sys.argv) < 2:
-print('Usage: %s' % (sys.argv[0]))
+import os, sys, argparse
+
+parser = argparse.ArgumentParser(description='Pulsar configuration file
customizer based on environment variables')
+parser.add_argument('--prefix', default='PULSAR_PREFIX_', help='Prefix for
environment variables, default is PULSAR_PREFIX_')
+parser.add_argument('conf_files', nargs='*', help='Configuration files')
+args = parser.parse_args()
+if not args.conf_files:
+parser.print_help()
(pulsar) branch branch-3.0 updated: [fix][broker] Sanitize values before logging in apply-config-from-env.py script (#22044)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new a88f592a759 [fix][broker] Sanitize values before logging in
apply-config-from-env.py script (#22044)
a88f592a759 is described below
commit a88f592a75964f6d899dd517cd01d29a653d4abe
Author: Lari Hotari
AuthorDate: Thu Feb 8 21:43:26 2024 -0800
[fix][broker] Sanitize values before logging in apply-config-from-env.py
script (#22044)
(cherry picked from commit 303678364eab538c16041214cae1588a5b2111d9)
---
.../scripts/apply-config-from-env-with-prefix.py | 85 ++
docker/pulsar/scripts/apply-config-from-env.py | 57 +++
2 files changed, 32 insertions(+), 110 deletions(-)
diff --git a/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
b/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
index 58f6c989750..9943b283a9f 100755
--- a/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
+++ b/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
@@ -32,83 +32,8 @@
# update if they exist and ignored if they don't.
-import os
-import sys
-
-if len(sys.argv) < 3:
-print('Usage: %s [...]' % (sys.argv[0]))
-sys.exit(1)
-
-# Always apply env config to env scripts as well
-prefix = sys.argv[1]
-conf_files = sys.argv[2:]
-
-PF_ENV_DEBUG = (os.environ.get('PF_ENV_DEBUG','0') == '1')
-
-for conf_filename in conf_files:
-lines = [] # List of config file lines
-keys = {} # Map a key to its line number in the file
-
-# Load conf file
-for line in open(conf_filename):
-lines.append(line)
-line = line.strip()
-if not line or line.startswith('#'):
-continue
-
-try:
-k,v = line.split('=', 1)
-keys[k] = len(lines) - 1
-except:
-if PF_ENV_DEBUG:
-print("[%s] skip Processing %s" % (conf_filename, line))
-
-# Update values from Env
-for k in sorted(os.environ.keys()):
-v = os.environ[k].strip()
-
-# Hide the value in logs if is password.
-if "password" in k.lower():
-displayValue = ""
-else:
-displayValue = v
-
-if k.startswith(prefix):
-k = k[len(prefix):]
-if k in keys:
-print('[%s] Applying config %s = %s' % (conf_filename, k,
displayValue))
-idx = keys[k]
-lines[idx] = '%s=%s\n' % (k, v)
-
-
-# Ensure we have a new-line at the end of the file, to avoid issue
-# when appending more lines to the config
-lines.append('\n')
-
-# Add new keys from Env
-for k in sorted(os.environ.keys()):
-v = os.environ[k]
-if not k.startswith(prefix):
-continue
-
-# Hide the value in logs if is password.
-if "password" in k.lower():
-displayValue = ""
-else:
-displayValue = v
-
-k = k[len(prefix):]
-if k not in keys:
-print('[%s] Adding config %s = %s' % (conf_filename, k,
displayValue))
-lines.append('%s=%s\n' % (k, v))
-else:
-print('[%s] Updating config %s = %s' % (conf_filename, k,
displayValue))
-lines[keys[k]] = '%s=%s\n' % (k, v)
-
-
-# Store back the updated config in the same file
-f = open(conf_filename, 'w')
-for line in lines:
-f.write(line)
-f.close()
+# DEPRECATED: Use "apply-config-from-env.py --prefix MY_PREFIX_ conf_file"
instead
+# this is not a python script, but a bash script. Call
apply-config-from-env.py with the prefix argument
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
+"${SCRIPT_DIR}/apply-config-from-env.py" --prefix "$1" "${@:2}"
diff --git a/docker/pulsar/scripts/apply-config-from-env.py
b/docker/pulsar/scripts/apply-config-from-env.py
index b8b479fc15b..da51f05f8be 100755
--- a/docker/pulsar/scripts/apply-config-from-env.py
+++ b/docker/pulsar/scripts/apply-config-from-env.py
@@ -25,18 +25,29 @@
## ./apply-config-from-env file.conf
##
-import os, sys
-
-if len(sys.argv) < 2:
-print('Usage: %s' % (sys.argv[0]))
+import os, sys, argparse
+
+parser = argparse.ArgumentParser(description='Pulsar configuration file
customizer based on environment variables')
+parser.add_argument('--prefix', default='PULSAR_PREFIX_', help='Prefix for
environment variables, default is PULSAR_PREFIX_')
+parser.add_argument('conf_files', nargs='*', help='Configuration files')
+args = parser.parse_args()
+if not args.conf_files:
+parser.print_help()
(pulsar) branch branch-2.10 updated: [fix][broker] Sanitize values before logging in apply-config-from-env.py script (#22044)
This is an automated email from the ASF dual-hosted git repository.
lhotari pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.10 by this push:
new f5922bc2730 [fix][broker] Sanitize values before logging in
apply-config-from-env.py script (#22044)
f5922bc2730 is described below
commit f5922bc2730bac2c455bb9866f1ba581897abd4f
Author: Lari Hotari
AuthorDate: Thu Feb 8 21:43:26 2024 -0800
[fix][broker] Sanitize values before logging in apply-config-from-env.py
script (#22044)
(cherry picked from commit 303678364eab538c16041214cae1588a5b2111d9)
---
.../scripts/apply-config-from-env-with-prefix.py | 99 --
docker/pulsar/scripts/apply-config-from-env.py | 57 ++---
2 files changed, 45 insertions(+), 111 deletions(-)
diff --git a/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
b/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
index 3f6bc2e4d3b..9943b283a9f 100755
--- a/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
+++ b/docker/pulsar/scripts/apply-config-from-env-with-prefix.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
@@ -18,85 +18,22 @@
# under the License.
#
-##
-## Edit a properties config file and replace values based on
-## the ENV variables
-## export prefix_my-key=new-value
-## ./apply-config-from-env-with-prefix prefix_ file.conf
-##
-
-import os, sys
-
-if len(sys.argv) < 3:
-print('Usage: %s' % (sys.argv[0]))
-sys.exit(1)
-
-# Always apply env config to env scripts as well
-prefix = sys.argv[1]
-conf_files = sys.argv[2:]
-
-PF_ENV_DEBUG = (os.environ.get('PF_ENV_DEBUG','0') == '1')
-
-for conf_filename in conf_files:
-lines = [] # List of config file lines
-keys = {} # Map a key to its line number in the file
-
-# Load conf file
-for line in open(conf_filename):
-lines.append(line)
-line = line.strip()
-if not line or line.startswith('#'):
-continue
-
-try:
-k,v = line.split('=', 1)
-keys[k] = len(lines) - 1
-except:
-if PF_ENV_DEBUG:
-print("[%s] skip Processing %s" % (conf_filename, line))
-
-# Update values from Env
-for k in sorted(os.environ.keys()):
-v = os.environ[k].strip()
-
-# Hide the value in logs if is password.
-if "password" in k.lower():
-displayValue = ""
-else:
-displayValue = v
-
-if k.startswith(prefix):
-k = k[len(prefix):]
-if k in keys:
-print('[%s] Applying config %s = %s' % (conf_filename, k,
displayValue))
-idx = keys[k]
-lines[idx] = '%s=%s\n' % (k, v)
-
-
-# Add new keys from Env
-for k in sorted(os.environ.keys()):
-v = os.environ[k]
-if not k.startswith(prefix):
-continue
-
-# Hide the value in logs if is password.
-if "password" in k.lower():
-displayValue = ""
-else:
-displayValue = v
-
-k = k[len(prefix):]
-if k not in keys:
-print('[%s] Adding config %s = %s' % (conf_filename, k,
displayValue))
-lines.append('%s=%s\n' % (k, v))
-else:
-print('[%s] Updating config %s = %s' % (conf_filename, k,
displayValue))
-lines[keys[k]] = '%s=%s\n' % (k, v)
-
+
+# Edit a properties config file and replace values based on
+# the ENV variables
+# export prefix_my-key=new-value
+# ./apply-config-from-env-with-prefix prefix_ file.conf
+#
+# Environment variables that are prefixed with the command
+# line prefix will be used to updated file properties if
+# they exist and create new ones if they don't.
+#
+# Environment variables not prefixed will be used only to
+# update if they exist and ignored if they don't.
+
-# Store back the updated config in the same file
-f = open(conf_filename, 'w')
-for line in lines:
-f.write(line)
-f.close()
+# DEPRECATED: Use "apply-config-from-env.py --prefix MY_PREFIX_ conf_file"
instead
+# this is not a python script, but a bash script. Call
apply-config-from-env.py with the prefix argument
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
+"${SCRIPT_DIR}/apply-config-from-env.py" --prefix "$1" "${@:2}"
diff --git a/docker/pulsar/scripts/apply-config-from-env.py
b/docker/pulsar/scripts/apply-config-from-env.py
index a802ca6fdbe..269d3fd221f 100755
--- a/docker/pulsar/scripts/apply-config-from-env.py
+++ b/docker/pulsar/scripts/apply-config-from-env.py
@@ -25,18 +25,29 @@
## ./apply-config-from-env
