[pulsar] branch master updated: [Build] Refactor integration test commands to run_integration_group.sh script (#9770)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new fc5ae1d [Build] Refactor integration test commands to run_integration_group.sh script (#9770) fc5ae1d is described below commit fc5ae1d454dd96ecfbe0d453e1ddf5258097d520 Author: Lari Hotari AuthorDate: Tue Mar 2 12:20:29 2021 +0200 [Build] Refactor integration test commands to run_integration_group.sh script (#9770) ### Motivation - Add a similar solution for running integration tests as there is for running unit tests in run_unit_group.sh script This benefit of this change is that it will be easier to refactor the GitHub Actions workflow build when the commands to run the integration tests are maintained in a separate bash script. ### Modifications - Move commands to run integration tests from the GitHub Actions workflow yaml files to a bash script `build/run_ingration_group.sh`. - The commands aren't modified as part of this PR, only moved to the script. --- .../ci-integration-backwards-compatibility.yaml| 2 +- .github/workflows/ci-integration-cli.yaml | 8 +- .../workflows/ci-integration-function-state.yaml | 2 +- .github/workflows/ci-integration-messaging.yaml| 12 +- .github/workflows/ci-integration-process.yaml | 12 +- .github/workflows/ci-integration-schema.yaml | 2 +- .github/workflows/ci-integration-sql.yaml | 2 +- .github/workflows/ci-integration-standalone.yaml | 2 +- .github/workflows/ci-integration-thread.yaml | 10 +- .../ci-integration-tiered-filesystem.yaml | 2 +- .../workflows/ci-integration-tiered-jcloud.yaml| 2 +- .github/workflows/ci-integration-transaction.yaml | 2 +- .github/workflows/ci-shade-test.yaml | 2 +- build/run_integration_group.sh | 131 + 14 files changed, 147 insertions(+), 44 deletions(-) diff --git a/.github/workflows/ci-integration-backwards-compatibility.yaml b/.github/workflows/ci-integration-backwards-compatibility.yaml index 4fc4e1a..f9fe802 100644 --- a/.github/workflows/ci-integration-backwards-compatibility.yaml +++ b/.github/workflows/ci-integration-backwards-compatibility.yaml @@ -90,4 +90,4 @@ jobs: - name: run integration tests if: steps.docs.outputs.changed_only == 'no' -run: ./build/retry.sh mvn -B -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-backwards-compatibility.xml -DintegrationTests -DredirectTestOutputToFile=false +run: ./build/run_integration_group.sh BACKWARDS_COMPAT diff --git a/.github/workflows/ci-integration-cli.yaml b/.github/workflows/ci-integration-cli.yaml index 7488b55..3500956 100644 --- a/.github/workflows/ci-integration-cli.yaml +++ b/.github/workflows/ci-integration-cli.yaml @@ -88,11 +88,7 @@ jobs: if: steps.docs.outputs.changed_only == 'no' run: mvn -B -f tests/docker-images/pom.xml install -am -Pdocker -DskipTests - - name: run pulsar cli integration tests + - name: run integration tests if: steps.docs.outputs.changed_only == 'no' -run: mvn -B -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-cli.xml -DintegrationTests -DredirectTestOutputToFile=false - - - name: run pulsar auth integration tests -if: steps.docs.outputs.changed_only == 'no' -run: mvn -B -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-auth.xml -DintegrationTests -DredirectTestOutputToFile=false +run: ./build/run_integration_group.sh CLI diff --git a/.github/workflows/ci-integration-function-state.yaml b/.github/workflows/ci-integration-function-state.yaml index 9f47c8b..6595f10 100644 --- a/.github/workflows/ci-integration-function-state.yaml +++ b/.github/workflows/ci-integration-function-state.yaml @@ -90,4 +90,4 @@ jobs: - name: run integration tests if: steps.docs.outputs.changed_only == 'no' -run: mvn -B -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-function-state.xml -DintegrationTests -DredirectTestOutputToFile=false +run: ./build/run_integration_group.sh FUNCTION_STATE \ No newline at end of file diff --git a/.github/workflows/ci-integration-messaging.yaml b/.github/workflows/ci-integration-messaging.yaml index 6b13630..1decf8b 100644 --- a/.github/workflows/ci-integration-messaging.yaml +++ b/.github/workflows/ci-integration-messaging.yaml @@ -88,14 +88,6 @@ jobs: if: steps.docs.outputs.changed_only == 'no' run: mvn -B -f tests/docker-images/pom.xml install -am -Pdocker -DskipTests -Ddocker.nocache=true - - name: run integration messaging tests + - name: run integration tests if: steps.docs.outputs.changed_only == 'no' -run: mvn -B -f tests/pom.xml test
[pulsar] branch master updated (935ceda -> fa4c7f3)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 935ceda [C++] SinglePartition message router is always picking the same partition (#9702) add fa4c7f3 [Docs] Fix url scheme typo (#9733) No new revisions were added by this update. Summary of changes: site2/docs/client-libraries-go.md | 2 +- site2/docs/security-oauth2.md | 4 ++-- site2/website/versioned_docs/version-2.6.1/client-libraries-go.md | 2 +- site2/website/versioned_docs/version-2.6.1/security-oauth2.md | 2 +- site2/website/versioned_docs/version-2.6.2/client-libraries-go.md | 2 +- site2/website/versioned_docs/version-2.6.2/security-oauth2.md | 2 +- site2/website/versioned_docs/version-2.6.3/client-libraries-go.md | 2 +- site2/website/versioned_docs/version-2.6.3/security-oauth2.md | 2 +- site2/website/versioned_docs/version-2.7.0/client-libraries-go.md | 2 +- site2/website/versioned_docs/version-2.7.0/security-oauth2.md | 4 ++-- site2/website/versioned_docs/version-2.7.1/client-libraries-go.md | 2 +- site2/website/versioned_docs/version-2.7.1/security-oauth2.md | 4 ++-- 12 files changed, 15 insertions(+), 15 deletions(-)
[pulsar] branch master updated (f5b1cb5 -> 590a27c)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from f5b1cb5 build with unix lineEnding in windows (#9536) add 590a27c [Issue 9549] Add friendly comment for topic compactor tool. (#9563) No new revisions were added by this update. Summary of changes: .../java/org/apache/pulsar/compaction/CompactorTool.java| 13 + 1 file changed, 13 insertions(+)
[pulsar] branch master updated (bd247a9 -> 40a1621)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from bd247a9 Adding Pandio as a user of Pulsar (#9534) add 40a1621 Remove usage of diff-only action with paths-ignore (#9527) No new revisions were added by this update. Summary of changes: .github/workflows/ci-cpp.yaml | 14 ++ .github/workflows/ci-go-functions-style.yaml | 10 -- .github/workflows/ci-go-functions-test.yaml| 8 .../ci-integration-backwards-compatibility.yaml| 16 ++-- .github/workflows/ci-integration-cli.yaml | 17 ++--- .github/workflows/ci-integration-function-state.yaml | 16 ++-- .github/workflows/ci-integration-messaging.yaml| 18 ++ .github/workflows/ci-integration-process.yaml | 18 ++ .github/workflows/ci-integration-schema.yaml | 16 ++-- .github/workflows/ci-integration-sql.yaml | 16 ++-- .github/workflows/ci-integration-standalone.yaml | 16 ++-- .github/workflows/ci-integration-thread.yaml | 18 ++ .../workflows/ci-integration-tiered-filesystem.yaml| 16 ++-- .github/workflows/ci-integration-tiered-jcloud.yaml| 16 ++-- .github/workflows/ci-integration-transaction.yaml | 16 ++-- .github/workflows/ci-license.yaml | 14 ++ .github/workflows/ci-shade-test.yaml | 16 ++-- .github/workflows/ci-unit-broker-broker-gp1.yaml | 12 ++-- .github/workflows/ci-unit-broker-broker-gp2.yaml | 11 ++- .github/workflows/ci-unit-broker-client-api.yaml | 12 ++-- .github/workflows/ci-unit-broker-client-impl.yaml | 12 ++-- .github/workflows/ci-unit-broker-other.yaml| 11 ++- .github/workflows/ci-unit-proxy.yaml | 11 ++- .github/workflows/ci-unit.yaml | 10 ++ 24 files changed, 44 insertions(+), 296 deletions(-)
[pulsar-test-infra] branch master updated: Set 'changed_only' to 'no' if commits cannot be found in diff-only action (#16)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-test-infra.git
The following commit(s) were added to refs/heads/master by this push:
new a3014b8 Set 'changed_only' to 'no' if commits cannot be found in
diff-only action (#16)
a3014b8 is described below
commit a3014b8784164dbc0aa39c42ae5422942c1e2f02
Author: Lari Hotari
AuthorDate: Tue Feb 9 04:13:05 2021 +0200
Set 'changed_only' to 'no' if commits cannot be found in diff-only action
(#16)
Fixes #12 https://github.com/apache/pulsar/issues/9526
The current solution causes the build to skip tests in Pulsar builds when
the pull request commits cannot be found. It's better that diff-only script
sets 'changed_only' to 'no' when it's not able to determine the correct result.
The commit for a PR won't be found when shallow clones are used and there
are more commits in the PR than the depth of the shallow clone. The Pulsar
GitHub Action workflows use shallow clone with the depth of 25 commits. This PR
will support such PRs by defaulting to `changed_only` to `no` when the commit
cannot be found.
This PR also fixes support for non-PR events such as `push` event.
`changed_only` is set to `no` when the triggering event isn't a `pull_request`
event.
---
diff-only/entrypoint.sh | 57 +
1 file changed, 34 insertions(+), 23 deletions(-)
diff --git a/diff-only/entrypoint.sh b/diff-only/entrypoint.sh
index 1b5f5d1..e3b94e3 100755
--- a/diff-only/entrypoint.sh
+++ b/diff-only/entrypoint.sh
@@ -9,30 +9,41 @@ echo "COMMITS: ${COMMITS}"
git --version
git rev-parse --abbrev-ref HEAD
-CHANGED_DIRS=$(git diff --name-only HEAD~${COMMITS} | awk -F "/*[^/]*/*$" '{
print ($1 == "" ? "." : $1); }' | sort | uniq)
-echo "CHANGED_DIRS are : ${CHANGED_DIRS}"
+if [[ $COMMITS -gt 0 ]]; then
+FIRST_COMMIT=$(git rev-parse HEAD~${COMMITS} 2> /dev/null)
+if [ $? -eq 0 ]; then
+CHANGED_DIRS=$(git diff --name-only $FIRST_COMMIT | awk -F
"/*[^/]*/*$" '{ print ($1 == "" ? "." : $1); }' | sort | uniq)
+echo "CHANGED_DIRS are : ${CHANGED_DIRS}"
-found_changed_dir_not_in_target_dirs="no"
-for changed_dir in ${CHANGED_DIRS}
-do
-matched="no"
-for target_dir in "${TARGET_DIRS[@]}"
-do
-if [[ ${changed_dir} == "${target_dir}"* ]]; then
-matched="yes"
-break
+found_changed_dir_not_in_target_dirs="no"
+for changed_dir in ${CHANGED_DIRS}
+do
+matched="no"
+for target_dir in "${TARGET_DIRS[@]}"
+do
+if [[ ${changed_dir} == "${target_dir}"* ]]; then
+matched="yes"
+break
+fi
+done
+if [[ ${matched} == "no" ]]; then
+found_changed_dir_not_in_target_dirs="yes"
+break
+fi
+done
+
+if [[ ${found_changed_dir_not_in_target_dirs} == "yes" ]]; then
+echo "Changes ${CHANGED_DIRS} not only in $*, setting
'changed_only' to 'no'"
+echo ::set-output name=changed_only::no
+else
+echo "Changes ${CHANGED_DIRS} only in $*, setting 'changed_only'
to 'yes'"
+echo ::set-output name=changed_only::yes
fi
-done
-if [[ ${matched} == "no" ]]; then
-found_changed_dir_not_in_target_dirs="yes"
-break
+else
+echo "Cannot find first commit. Setting 'changed_only' to 'no'."
+echo ::set-output name=changed_only::no
fi
-done
-
-if [[ ${found_changed_dir_not_in_target_dirs} == "yes" ]]; then
-echo "Changes ${CHANGED_DIRS} not only in $*, setting 'changed_only' to
'no'"
-echo ::set-output name=changed_only::no
else
-echo "Changes ${CHANGED_DIRS} only in $*, setting 'changed_only' to 'yes'"
-echo ::set-output name=changed_only::yes
-fi
+echo "Cannot find number of commits in pull_request. Setting
'changed_only' to 'no'."
+echo ::set-output name=changed_only::no
+fi
\ No newline at end of file
[pulsar] branch master updated (b3dc142 -> e006165)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from b3dc142 add missing comma (#9429) add e006165 [Java Client] Batch version of acknowledgeAsync will return a completed future before individual messages complete (#9383) No new revisions were added by this update. Summary of changes: .../client/api/ConsumerBatchReceiveTest.java | 231 + .../apache/pulsar/client/impl/ConsumerBase.java| 8 +- .../PersistentAcknowledgmentsGroupingTracker.java | 8 +- 3 files changed, 197 insertions(+), 50 deletions(-)
[pulsar] branch master updated (ebd70de -> a1107ad)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from ebd70de Fix setting backlogQuota will always succeed (#9382) add a1107ad [PIP-71][SQL]Pulsar SQL migrate SchemaHandle to presto decoder (#8422) No new revisions were added by this update. Summary of changes: pulsar-sql/presto-distribution/LICENSE | 3 + pulsar-sql/presto-pulsar/pom.xml | 26 ++ .../pulsar/sql/presto/AvroSchemaHandler.java | 101 - .../pulsar/sql/presto/JSONSchemaHandler.java | 108 - .../pulsar/sql/presto/KeyValueSchemaHandler.java | 105 - .../pulsar/sql/presto/PulsarColumnHandle.java | 75 ++-- .../pulsar/sql/presto/PulsarColumnMetadata.java| 128 +- .../pulsar/sql/presto/PulsarConnectorModule.java | 8 + .../presto/PulsarDispatchingRowDecoderFactory.java | 84 .../sql/presto/PulsarFieldValueProviders.java | 47 +- .../pulsar/sql/presto/PulsarInternalColumn.java| 173 ++-- .../apache/pulsar/sql/presto/PulsarMetadata.java | 279 ++-- .../sql/presto/PulsarPrimitiveSchemaHandler.java | 61 --- .../pulsar/sql/presto/PulsarRecordCursor.java | 303 +++-- .../apache/pulsar/sql/presto/PulsarRecordSet.java | 10 +- .../pulsar/sql/presto/PulsarRecordSetProvider.java | 8 +- .../apache/pulsar/sql/presto/PulsarRowDecoder.java | 23 +- .../pulsar/sql/presto/PulsarRowDecoderFactory.java | 51 +++ .../pulsar/sql/presto/PulsarSchemaHandlers.java| 67 --- .../pulsar/sql/presto/PulsarSplitManager.java | 2 +- .../sql/presto/PulsarSqlSchemaInfoProvider.java| 13 +- .../apache/pulsar/sql/presto/SchemaHandler.java| 45 -- .../decoder/avro/PulsarAvroColumnDecoder.java | 385 + .../presto/decoder/avro/PulsarAvroRowDecoder.java | 77 .../decoder/avro/PulsarAvroRowDecoderFactory.java | 190 + .../sql/presto/decoder/avro}/package-info.java | 5 +- .../decoder/json/PulsarJsonFieldDecoder.java | 438 +++ .../presto/decoder/json/PulsarJsonRowDecoder.java | 89 .../decoder/json/PulsarJsonRowDecoderFactory.java | 186 .../sql/presto/decoder/json}/package-info.java | 5 +- .../primitive/PulsarPrimitiveRowDecoder.java | 101 + .../PulsarPrimitiveRowDecoderFactory.java | 115 + .../presto/decoder/primitive}/package-info.java| 5 +- .../pulsar/sql/presto/TestAvroSchemaHandler.java | 127 -- .../pulsar/sql/presto/TestPulsarConnector.java | 471 + .../sql/presto/TestPulsarConnectorConfig.java | 2 +- .../presto/TestPulsarKeyValueSchemaHandler.java| 353 --- .../pulsar/sql/presto/TestPulsarMetadata.java | 56 +-- .../presto/TestPulsarPrimitiveSchemaHandler.java | 164 --- .../pulsar/sql/presto/TestPulsarRecordCursor.java | 391 + .../sql/presto/decoder/AbstractDecoderTester.java | 133 ++ .../sql/presto/decoder/DecoderTestMessage.java | 91 .../pulsar/sql/presto/decoder/DecoderTestUtil.java | 121 ++ .../presto/decoder/avro/AvroDecoderTestUtil.java | 199 + .../sql/presto/decoder/avro/TestAvroDecoder.java | 290 + .../presto/decoder/json/JsonDecoderTestUtil.java | 199 + .../sql/presto/decoder/json/TestJsonDecoder.java | 284 + .../primitive/PrimitiveDecoderTestUtil.java| 36 +- .../decoder/primitive/TestPrimitiveDecoder.java| 233 ++ 49 files changed, 4194 insertions(+), 2272 deletions(-) delete mode 100644 pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/AvroSchemaHandler.java delete mode 100644 pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/JSONSchemaHandler.java delete mode 100644 pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/KeyValueSchemaHandler.java create mode 100644 pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarDispatchingRowDecoderFactory.java copy managed-ledger/src/main/java/org/apache/bookkeeper/mledger/util/SafeRun.java => pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarFieldValueProviders.java (52%) delete mode 100644 pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarPrimitiveSchemaHandler.java copy pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/RecordWithEncryptionContext.java => pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarRowDecoder.java (65%) create mode 100644 pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarRowDecoderFactory.java delete mode 100644 pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarSchemaHandlers.java delete mode 100644 pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/
[pulsar] branch master updated (fa66a24 -> 88f8fa4)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from fa66a24 [pulsar-broker] broker resources use metadata-store api (#9346) add 88f8fa4 add logging config file (#9365) No new revisions were added by this update. Summary of changes: site2/docs/reference-configuration.md | 2 ++ 1 file changed, 2 insertions(+)
[pulsar-client-go] branch master updated: Clearing message queues after seek requests (#419)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-client-go.git
The following commit(s) were added to refs/heads/master by this push:
new c24191b Clearing message queues after seek requests (#419)
c24191b is described below
commit c24191b37a98d8731b34fb9fd44c151a2ef3a99e
Author: Miloš Matijašević
AuthorDate: Tue Jan 26 04:04:52 2021 +0100
Clearing message queues after seek requests (#419)
### Motivation
Message queues should be cleared after seek requests (`Seek` and
`SeekByTime`). If this is not performed messages that stay in message queues
will be consumed before the sough message.
### Modifications
Cleaning `queueCh` and `messageCh` after successful seek request in
partition_consumer.go.
### Verifying this change
- [x] Make sure that the change passes the CI checks.
This change is already covered by existing tests, such as
`TestConsumerSeekByTime` and `TestConsumerSeek`.
This change added tests and can be verified as follows:
- Extended `TestConsumerSeekByTime` and `TestConsumerSeek` test for
consuming correctly sought message even if some messages stay in message queues.
Signed-off-by: milos-matijasevic
---
pulsar/consumer_partition.go | 42 +-
pulsar/consumer_test.go | 20
2 files changed, 49 insertions(+), 13 deletions(-)
diff --git a/pulsar/consumer_partition.go b/pulsar/consumer_partition.go
index d970f65..a787ffb 100644
--- a/pulsar/consumer_partition.go
+++ b/pulsar/consumer_partition.go
@@ -108,11 +108,12 @@ type partitionConsumer struct {
startMessageID trackingMessageID
lastDequeuedMsg trackingMessageID
- eventsChchan interface{}
- connectedCh chan struct{}
- connectClosedCh chan connectionClosed
- closeCh chan struct{}
- clearQueueChchan func(id trackingMessageID)
+ eventsCh chan interface{}
+ connectedCh chan struct{}
+ connectClosedCh chan connectionClosed
+ closeCh chan struct{}
+ clearQueueCh chan func(id trackingMessageID)
+ clearMessageQueuesCh chan chan struct{}
nackTracker *negativeAcksTracker
dlq *dlqRouter
@@ -144,6 +145,7 @@ func newPartitionConsumer(parent Consumer, client *client,
options *partitionCon
connectClosedCh: make(chan connectionClosed, 10),
closeCh: make(chan struct{}),
clearQueueCh: make(chan func(id trackingMessageID)),
+ clearMessageQueuesCh: make(chan chan struct{}),
compressionProviders:
make(map[pb.CompressionType]compression.Provider),
dlq: dlq,
metrics: metrics,
@@ -361,6 +363,7 @@ func (pc *partitionConsumer) requestSeek(msgID messageID)
error {
pc.log.WithError(err).Error("Failed to reset to message id")
return err
}
+ pc.clearMessageChannels()
return nil
}
@@ -395,7 +398,15 @@ func (pc *partitionConsumer) internalSeekByTime(seek
*seekByTimeRequest) {
if err != nil {
pc.log.WithError(err).Error("Failed to reset to message publish
time")
seek.err = err
+ return
}
+ pc.clearMessageChannels()
+}
+
+func (pc *partitionConsumer) clearMessageChannels() {
+ doneCh := make(chan struct{})
+ pc.clearMessageQueuesCh <- doneCh
+ <-doneCh
}
func (pc *partitionConsumer) internalAck(req *ackRequest) {
@@ -659,6 +670,27 @@ func (pc *partitionConsumer) dispatcher() {
}
clearQueueCb(nextMessageInQueue)
+
+ case doneCh := <-pc.clearMessageQueuesCh:
+ for len(pc.queueCh) > 0 {
+ <-pc.queueCh
+ }
+ for len(pc.messageCh) > 0 {
+ <-pc.messageCh
+ }
+ messages = nil
+
+ // reset available permits
+ pc.availablePermits = 0
+ initialPermits := uint32(pc.queueSize)
+
+ pc.log.Debugf("dispatcher requesting initial
permits=%d", initialPermits)
+ // send initial permits
+ if err := pc.internalFlow(initialPermits); err != nil {
+ pc.log.WithError(err).Error("unable to send
initial permits to broker")
+ }
+
+ close(doneCh)
}
}
}
diff --git a/pulsar/consumer_test.go b/pulsa
[pulsar] branch master updated (66576fa -> 39dc24b)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 66576fa Delete folders that do not need to be submitted (#9171) add 39dc24b fix un-correct tips for create topic error when namespace not exist (#9223) No new revisions were added by this update. Summary of changes: .../java/org/apache/pulsar/broker/admin/AdminResource.java | 3 +++ .../java/org/apache/pulsar/broker/admin/AdminApiTest.java| 12 2 files changed, 15 insertions(+)
[pulsar] branch master updated (17c0d11 -> 66576fa)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 17c0d11 [Spotbugs] Enable spotbugs in module pulsar-broker-auth-athenz and pulsar-client-auth-athenz. (#8857) add 66576fa Delete folders that do not need to be submitted (#9171) No new revisions were added by this update. Summary of changes: node_modules/.yarn-integrity | 10 -- 1 file changed, 10 deletions(-) delete mode 100644 node_modules/.yarn-integrity
[pulsar] branch master updated (7b65fab -> e9fd7f1)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 7b65fab Support dispatch rate policy at the topic level (#9175) add e9fd7f1 fix max topic for namespace does not work (#9193) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/pulsar/broker/admin/AdminResource.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[pulsar] branch master updated (8ceaf43 -> 527eb31)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 8ceaf43 fix format issue (#9158) add 527eb31 Upgrade Bouncy Castle to 1.68 (#9199) No new revisions were added by this update. Summary of changes: distribution/server/src/assemble/LICENSE.bin.txt | 6 +++--- pom.xml | 2 +- pulsar-sql/presto-distribution/LICENSE | 6 +++--- 3 files changed, 7 insertions(+), 7 deletions(-)
[pulsar] branch master updated (a526506 -> 7a9ec06)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.
from a526506 Update Presto (now Trino) links (#9104)
add 7a9ec06 Update Pulsar service URL description for Debezium connector
(#9100)
No new revisions were added by this update.
Summary of changes:
site2/docs/io-cdc-debezium.md | 2 +-
site2/website/versioned_docs/version-2.2.1/io-cdc.md | 2 +-
site2/website/versioned_docs/version-2.3.0/io-cdc-debezium.md | 2 +-
.../{version-2.3.0 => version-2.3.1}/io-cdc-debezium.md| 10 +-
.../{version-2.3.0 => version-2.3.2}/io-cdc-debezium.md| 10 +-
site2/website/versioned_docs/version-2.4.0/io-cdc-debezium.md | 2 +-
site2/website/versioned_docs/version-2.4.1/io-cdc-debezium.md | 2 +-
site2/website/versioned_docs/version-2.4.2/io-cdc-debezium.md | 2 +-
site2/website/versioned_docs/version-2.5.0/io-cdc-debezium.md | 2 +-
.../website/versioned_docs/version-2.5.1/io-debezium-source.md | 2 +-
.../website/versioned_docs/version-2.5.2/io-debezium-source.md | 2 +-
site2/website/versioned_docs/version-2.6.0/io-cdc-debezium.md | 2 +-
site2/website/versioned_docs/version-2.6.1/io-cdc-debezium.md | 2 +-
site2/website/versioned_docs/version-2.6.3/io-cdc-debezium.md | 2 +-
site2/website/versioned_docs/version-2.7.0/io-cdc-debezium.md | 2 +-
site2/website/versioned_docs/version-2.7.1/io-cdc-debezium.md | 2 +-
16 files changed, 24 insertions(+), 24 deletions(-)
copy site2/website/versioned_docs/{version-2.3.0 =>
version-2.3.1}/io-cdc-debezium.md (95%)
copy site2/website/versioned_docs/{version-2.3.0 =>
version-2.3.2}/io-cdc-debezium.md (95%)
[pulsar] branch master updated: Remove external CIs (#9069)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 331ca7c Remove external CIs (#9069)
331ca7c is described below
commit 331ca7c6845c9dc43cbfa458704c85540e6ae2a1
Author: lipenghui
AuthorDate: Mon Dec 28 19:21:36 2020 +0800
Remove external CIs (#9069)
Remove external CIs
https://lists.apache.org/thread.html/r900f8f9a874006ed8121bdc901a0d1acccbb340882c1f94dad61a5e9%40%3Cusers.infra.apache.org%3E
```
Hi folks,
As part of an ongoing effort to keep offers in line with our standard
practices and provenance policies, we have decided to only allow GitHub
Actions to be defined locally.
This is a global change, and thus affects all repositories. If you were
using externally defined actions, please be sure to bring them into your
local repository instead.
---
.github/workflows/ci-build-macos.yaml | 5 -
.github/workflows/ci-cpp.yaml | 5 -
.github/workflows/ci-go-functions-style.yaml | 5 -
.github/workflows/ci-go-functions-test.yaml | 5 -
.github/workflows/ci-integration-backwards-compatibility.yaml | 5 -
.github/workflows/ci-integration-cli.yaml | 5 -
.github/workflows/ci-integration-function-state.yaml | 5 -
.github/workflows/ci-integration-messaging.yaml | 5 -
.github/workflows/ci-integration-process.yaml | 5 -
.github/workflows/ci-integration-schema.yaml | 5 -
.github/workflows/ci-integration-sql.yaml | 5 -
.github/workflows/ci-integration-standalone.yaml | 5 -
.github/workflows/ci-integration-thread.yaml | 5 -
.github/workflows/ci-integration-tiered-filesystem.yaml | 5 -
.github/workflows/ci-integration-tiered-jcloud.yaml | 5 -
.github/workflows/ci-integration-transaction.yaml | 5 -
.github/workflows/ci-license.yaml | 5 -
.github/workflows/ci-pulsar-website-build.yaml| 5 -
.github/workflows/ci-shade-test.yaml | 5 -
.github/workflows/ci-unit-broker-broker-gp1.yaml | 5 -
.github/workflows/ci-unit-broker-broker-gp2.yaml | 5 -
.github/workflows/ci-unit-broker-client-api.yaml | 5 -
.github/workflows/ci-unit-broker-client-impl.yaml | 5 -
.github/workflows/ci-unit-broker-other.yaml | 5 -
.github/workflows/ci-unit-proxy.yaml | 5 -
.github/workflows/ci-unit.yaml| 5 -
26 files changed, 130 deletions(-)
diff --git a/.github/workflows/ci-build-macos.yaml
b/.github/workflows/ci-build-macos.yaml
index 2eb3a84..a7dd482 100644
--- a/.github/workflows/ci-build-macos.yaml
+++ b/.github/workflows/ci-build-macos.yaml
@@ -39,11 +39,6 @@ jobs:
timeout-minutes: 120
steps:
- - name: Cancel Previous Runs
-uses: styfle/cancel-workflow-action@0.6.0
-with:
- access_token: ${{ github.token }}
-
- name: checkout
uses: actions/checkout@v2
with:
diff --git a/.github/workflows/ci-cpp.yaml b/.github/workflows/ci-cpp.yaml
index e36c508..d4991e9 100644
--- a/.github/workflows/ci-cpp.yaml
+++ b/.github/workflows/ci-cpp.yaml
@@ -37,11 +37,6 @@ jobs:
timeout-minutes: 120
steps:
- - name: Cancel Previous Runs
-uses: styfle/cancel-workflow-action@0.6.0
-with:
- access_token: ${{ github.token }}
-
- name: checkout
uses: actions/checkout@v2
with:
diff --git a/.github/workflows/ci-go-functions-style.yaml
b/.github/workflows/ci-go-functions-style.yaml
index f7004fc..80d7e51 100644
--- a/.github/workflows/ci-go-functions-style.yaml
+++ b/.github/workflows/ci-go-functions-style.yaml
@@ -45,11 +45,6 @@ jobs:
go-version: [1.11, 1.12, 1.13, 1.14]
steps:
- - name: Cancel Previous Runs
-uses: styfle/cancel-workflow-action@0.6.0
-with:
- access_token: ${{ github.token }}
-
- name: Check out code into the Go module directory
uses: actions/checkout@v2
with:
diff --git a/.github/workflows/ci-go-functions-test.yaml
b/.github/workflows/ci-go-functions-test.yaml
index 5a015bd..fa7543b 100644
--- a/.github/workflows/ci-go-functions-test.yaml
+++ b/.github/workflows/ci-go-functions-test.yaml
@@ -47,11 +47,6 @@ jobs:
timeout-minutes: 120
steps:
- - name: Cancel Previous Runs
-uses: styfle/cancel-workflow-action@0.6.0
-with:
- access_token: ${{ github.token }}
-
- name: Check out code into the Go
[pulsar] branch master updated: make namespaces policy update take effect on time (#8976)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 3604c67 make namespaces policy update take effect on time (#8976)
3604c67 is described below
commit 3604c67c7196a4180f93ff441e105ea146584bfe
Author: Jia Zhai
AuthorDate: Mon Dec 21 13:59:17 2020 +0800
make namespaces policy update take effect on time (#8976)
### Motivation
The change of namespaces isolation policy takes effect only when
load-manager re-assign the bundles to brokers again.
This change tries to make the isolation policy takes effect on time.
### Modifications
- change setNamespaceIsolationPolicy method into async.
- add parameter to enable this feature:
enableNamespaceIsolationUpdateOnTime.
- add test to cover this feature.
### Verifying this change
tests passed
---
.../apache/pulsar/broker/ServiceConfiguration.java | 7 ++
.../pulsar/broker/admin/impl/ClustersBase.java | 113 +++--
.../pulsar/broker/loadbalance/LoadManager.java | 8 ++
.../broker/loadbalance/ModularLoadManager.java | 7 ++
.../loadbalance/impl/ModularLoadManagerImpl.java | 9 +-
.../impl/ModularLoadManagerWrapper.java| 5 +
.../apache/pulsar/broker/admin/AdminApiTest2.java | 57 +++
.../org/apache/pulsar/broker/admin/AdminTest.java | 3 +-
8 files changed, 199 insertions(+), 10 deletions(-)
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index 97799cd..d065c85 100644
---
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -882,6 +882,13 @@ public class ServiceConfiguration implements
PulsarConfiguration {
doc = "List of interceptors for entry metadata.")
private Set brokerEntryMetadataInterceptors = new HashSet<>();
+@FieldContext(
+category = CATEGORY_SERVER,
+doc = "Enable namespaceIsolation policy update take effect ontime or
not," +
+" if set to ture, then the related namespaces will be unloaded
after reset policy to make it take effect."
+)
+private boolean enableNamespaceIsolationUpdateOnTime = false;
+
/* --- TLS --- /
@FieldContext(
category = CATEGORY_TLS,
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/ClustersBase.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/ClustersBase.java
index 5e9dfbf..d2714ec 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/ClustersBase.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/ClustersBase.java
@@ -37,6 +37,8 @@ import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.Collectors;
import javax.ws.rs.DELETE;
import javax.ws.rs.GET;
@@ -44,12 +46,16 @@ import javax.ws.rs.POST;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
+import javax.ws.rs.container.AsyncResponse;
+import javax.ws.rs.container.Suspended;
import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.Status;
import org.apache.bookkeeper.util.ZkUtils;
import org.apache.pulsar.broker.admin.AdminResource;
import org.apache.pulsar.broker.cache.ConfigurationCacheService;
import org.apache.pulsar.broker.web.RestException;
+import org.apache.pulsar.client.admin.Namespaces;
import org.apache.pulsar.common.naming.Constants;
import org.apache.pulsar.common.naming.NamedEntity;
import org.apache.pulsar.common.policies.data.BrokerNamespaceIsolationData;
@@ -58,6 +64,7 @@ import org.apache.pulsar.common.policies.data.FailureDomain;
import org.apache.pulsar.common.policies.data.NamespaceIsolationData;
import org.apache.pulsar.common.policies.impl.NamespaceIsolationPolicies;
import org.apache.pulsar.common.policies.impl.NamespaceIsolationPolicyImpl;
+import org.apache.pulsar.common.util.FutureUtil;
import org.apache.pulsar.common.util.ObjectMapperFactory;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
@@ -675,6 +682,7 @@ public class ClustersBase extends AdminResource {
@ApiResponse(code = 500, message = "Internal server error.")
})
public void setNamespaceIsolationPolicy(
+@Suspended final AsyncResponse asyncResponse,
@ApiParam(
value = "The cluster name",
required = true
[pulsar] branch master updated (31f7d70 -> a292b0a)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 31f7d70 Allow stats operations not to be blocked in functions (#9005) add a292b0a Improve error handling when broker doesn't trust client certificates (#8998) No new revisions were added by this update. Summary of changes: .../apache/pulsar/broker/authentication/AuthenticationDataCommand.java | 3 +++ .../apache/pulsar/broker/authentication/AuthenticationProviderTls.java | 3 +++ 2 files changed, 6 insertions(+)
[pulsar] branch master updated: Enable pulsar-perf to load WebSocket service URL from conf file (#9000)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new b43e83c Enable pulsar-perf to load WebSocket service URL from conf
file (#9000)
b43e83c is described below
commit b43e83c87b7cb7a9e622a7001f33e75347df1c65
Author: Masahiro Sakamoto
AuthorDate: Sun Dec 20 22:25:50 2020 +0900
Enable pulsar-perf to load WebSocket service URL from conf file (#9000)
### Motivation
Currently, the command line option `--proxy-url` is required when running
the subcommand `websocket-producer` of `pulsar-perf`.
```sh
$ ./bin/pulsar-perf websocket-producer
The following option is required: -u, --proxy-url
Usage: pulsar-perf websocket-producer [options]
persistent://tenant/ns/my-topic
Options:
...
* -u, --proxy-url
Pulsar Proxy URL, e.g., "ws://localhost:8080/"
```
I think it would be useful to be able to load the WebSocket service URL
from the config file (such as `client.conf`) as well as other parameters.
### Modifications
`pulsar-perf` loads the WebSocket service URL from the config file if
`--proxy-url` is not specified. The priorities are as follows:
1. `--proxy-url` option
2. `webSocketServiceUrl` in the config file
3. `webServiceUrl` in the config file
4. `serviceUrl` in the config file
---
.../proxy/socket/client/PerformanceClient.java | 31 -
.../proxy/socket/client/PerformanceClientTest.java | 75 ++
.../src/test/resources/websocket_client1.conf | 22 +++
.../src/test/resources/websocket_client2.conf | 21 ++
.../src/test/resources/websocket_client3.conf | 20 ++
5 files changed, 166 insertions(+), 3 deletions(-)
diff --git
a/pulsar-testclient/src/main/java/org/apache/pulsar/proxy/socket/client/PerformanceClient.java
b/pulsar-testclient/src/main/java/org/apache/pulsar/proxy/socket/client/PerformanceClient.java
index 5253d63..9722677 100644
---
a/pulsar-testclient/src/main/java/org/apache/pulsar/proxy/socket/client/PerformanceClient.java
+++
b/pulsar-testclient/src/main/java/org/apache/pulsar/proxy/socket/client/PerformanceClient.java
@@ -18,6 +18,9 @@
*/
package org.apache.pulsar.proxy.socket.client;
+import static org.apache.commons.lang3.StringUtils.isBlank;
+import static org.apache.commons.lang3.StringUtils.isNotBlank;
+
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
@@ -72,7 +75,7 @@ public class PerformanceClient {
@Parameter(names = { "--conf-file" }, description = "Configuration
file")
public String confFile;
-@Parameter(names = { "-u", "--proxy-url" }, description = "Pulsar
Proxy URL, e.g., \"ws://localhost:8080/\"", required = true)
+@Parameter(names = { "-u", "--proxy-url" }, description = "Pulsar
Proxy URL, e.g., \"ws://localhost:8080/\"")
public String proxyURL;
@Parameter(description = "persistent://tenant/ns/my-topic", required =
true)
@@ -144,8 +147,22 @@ public class PerformanceClient {
System.exit(1);
}
-if (arguments.proxyURL == null) {
-arguments.proxyURL = prop.getProperty("serviceUrl",
"http://localhost:8080/;);
+if (isBlank(arguments.proxyURL)) {
+String webSocketServiceUrl =
prop.getProperty("webSocketServiceUrl");
+if (isNotBlank(webSocketServiceUrl)) {
+arguments.proxyURL = webSocketServiceUrl;
+} else {
+String webServiceUrl =
isNotBlank(prop.getProperty("webServiceUrl"))
+? prop.getProperty("webServiceUrl")
+: prop.getProperty("serviceUrl");
+if (isNotBlank(webServiceUrl)) {
+if (webServiceUrl.startsWith("ws://") ||
webServiceUrl.startsWith("wss://")) {
+arguments.proxyURL = webServiceUrl;
+} else if (webServiceUrl.startsWith("http://;) ||
webServiceUrl.startsWith("https://;)) {
+arguments.proxyURL =
webServiceUrl.replaceFirst("^http", "ws");
+}
+}
+}
}
if (arguments.authPluginClassName == null) {
@@ -157,6 +174,14 @@ public class PerformanceClient {
}
}
+if (isBlank(arguments.proxyURL)) {
+arguments.proxyURL = "ws://localhost:8080/";
+}
+
+if (!arguments
[pulsar] branch master updated (5f9ecb5 -> 1966d68)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 5f9ecb5 Update schema type (#9003) add 1966d68 update the Woraround in versioned docs (#9002) No new revisions were added by this update. Summary of changes: .../versioned_docs/version-2.4.0/functions-worker.md| 13 + .../versioned_docs/version-2.4.1/functions-worker.md| 13 + .../versioned_docs/version-2.4.2/functions-worker.md| 13 + .../versioned_docs/version-2.5.0/functions-worker.md| 13 + .../versioned_docs/version-2.6.0/functions-worker.md| 13 + .../versioned_docs/version-2.6.1/functions-worker.md| 13 + .../versioned_docs/version-2.6.2/functions-worker.md| 13 + .../versioned_docs/version-2.7.0/functions-worker.md| 13 + 8 files changed, 72 insertions(+), 32 deletions(-)
[pulsar] branch master updated: Update schema type (#9003)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 5f9ecb5 Update schema type (#9003) 5f9ecb5 is described below commit 5f9ecb5e2c349aa7190ced3a34ddb7c3d3c84c61 Author: HuanliMeng <48120384+huanli-m...@users.noreply.github.com> AuthorDate: Sun Dec 20 21:08:09 2020 +0800 Update schema type (#9003) ### Motivation In PR #7874, Instant, LocalDate, LocalTime, LocalDateTime are added to Pulsar's primitive schemas codes. But doc is not updated accordingly. ### Modifications Update the Pulsar docs to support Instant, LocalDate, LocalTime, LocalDateTime for Pulsar's primitive schemas. - Affected docs: Understand schema - sections: primitive type. Affected releases: master and 2.7.0 --- site2/docs/schema-understand.md | 8 site2/website/versioned_docs/version-2.7.0/schema-understand.md | 8 2 files changed, 16 insertions(+) diff --git a/site2/docs/schema-understand.md b/site2/docs/schema-understand.md index 86bacc9..7d26221 100644 --- a/site2/docs/schema-understand.md +++ b/site2/docs/schema-understand.md @@ -143,6 +143,10 @@ Currently, Pulsar supports the following primitive types: | `BYTES` | A sequence of 8-bit unsigned bytes | | `STRING` | A Unicode character sequence | | `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | +| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| +| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| +| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| +| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | For primitive types, Pulsar does not store any schema data in `SchemaInfo`. The `type` in `SchemaInfo` is used to determine how to serialize and deserialize the data. @@ -164,6 +168,10 @@ The conversions between **Pulsar schema types** and **language-specific primitiv | TIMESTAMP | java.sql.Timestamp | | | | TIME | java.sql.Time | | | | DATE | java.util.Date | | | +| INSTANT | java.time.Instant | | | +| LOCAL_DATE | java.time.LocalDate | | | +| LOCAL_TIME | java.time.LocalDateTime | | +| LOCAL_DATE_TIME | java.time.LocalTime | | **Example** diff --git a/site2/website/versioned_docs/version-2.7.0/schema-understand.md b/site2/website/versioned_docs/version-2.7.0/schema-understand.md index 051d81a..f81d97e 100644 --- a/site2/website/versioned_docs/version-2.7.0/schema-understand.md +++ b/site2/website/versioned_docs/version-2.7.0/schema-understand.md @@ -144,6 +144,10 @@ Currently, Pulsar supports the following primitive types: | `BYTES` | A sequence of 8-bit unsigned bytes | | `STRING` | A Unicode character sequence | | `TIMESTAMP` (`DATE`, `TIME`) | A logic type represents a specific instant in time with millisecond precision. It stores the number of milliseconds since `January 1, 1970, 00:00:00 GMT` as an `INT64` value | +| INSTANT | A single instantaneous point on the time-line with nanoseconds precision| +| LOCAL_DATE | An immutable date-time object that represents a date, often viewed as year-month-day| +| LOCAL_TIME | An immutable date-time object that represents a time, often viewed as hour-minute-second. Time is represented to nanosecond precision.| +| LOCAL_DATE_TIME | An immutable date-time object that represents a date-time, often viewed as year-month-day-hour-minute-second | For primitive types, Pulsar does not store any schema data in `SchemaInfo`. The `type` in `SchemaInfo` is used to determine how to serialize and deserialize the data. @@ -165,6 +169,10 @@ The conversions between **Pulsar schema types** and **language-specific primitiv | TIMESTAMP | java.sql.Timestamp | | | | TIME | java.sql.Time | | | | DATE | java.util.Date | | | +| INSTANT | java.time.Instant | | | +| LOCAL_DATE | java.time.LocalDate | | | +| LOCAL_TIME | java.time.LocalDateTime | | +| LOCAL_DATE_TIME | java.time.LocalTime | | **Example**
[pulsar] branch master updated (6275297 -> a1d814b)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 6275297 [PIP 70][Issue 8617] Introduce lightweight broker entry metadata (#8618) add a1d814b Fix pulsar-broker style error (#8933) No new revisions were added by this update. Summary of changes: .../apache/pulsar/broker/namespace/NamespaceService.java | 8 +--- .../pulsar/tests/integration/presto/TestBasicPresto.java | 16 ++-- 2 files changed, 19 insertions(+), 5 deletions(-)
[pulsar] branch master updated (b63e288 -> 6275297)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.
from b63e288 Support configure max subscriptions per topic on the
namespace level policy (#8924)
add 6275297 [PIP 70][Issue 8617] Introduce lightweight broker entry
metadata (#8618)
No new revisions were added by this update.
Summary of changes:
.../bookkeeper/mledger/ManagedLedgerConfig.java| 1 +
.../apache/bookkeeper/mledger/impl/OpAddEntry.java | 7 +-
.../apache/pulsar/broker/ServiceConfiguration.java | 6 +
.../pulsar/broker/service/BrokerService.java | 17 +
.../broker/service/BrokerServiceException.java | 6 +
.../broker/service/PulsarCommandSenderImpl.java| 2 +
.../apache/pulsar/broker/service/ServerCnx.java| 5 +
.../nonpersistent/NonPersistentReplicator.java | 2 +-
.../persistent/PersistentMessageExpiryMonitor.java | 4 +-
.../persistent/PersistentMessageFinder.java| 6 +-
.../service/persistent/PersistentReplicator.java | 2 +-
.../broker/service/persistent/PersistentTopic.java | 34 +-
.../service/PersistentMessageFinderTest.java | 126 +++
.../org/apache/pulsar/client/impl/MessageImpl.java | 73 +++-
.../apache/pulsar/common/api/proto/PulsarApi.java | 380 +
.../AppendBrokerTimestampMetadataInterceptor.java | 15 +-
...on.java => BrokerEntryMetadataInterceptor.java} | 17 +-
.../common/intercept/BrokerEntryMetadataUtils.java | 58
.../apache/pulsar/common/protocol/Commands.java| 79 +
pulsar-common/src/main/proto/PulsarApi.proto | 7 +
.../pulsar/common/protocol/CommandUtilsTests.java | 104 ++
.../pulsar/sql/presto/PulsarSplitManager.java | 9 +-
tests/{bc_2_0_1 => bc_2_6_0}/pom.xml | 25 +-
.../pulsar/tests/integration/PulsarContainer.java | 0
.../apache/pulsar/tests/integration/SmokeTest.java | 0
.../src/test/resources/backwards-client.xml} | 0
tests/pom.xml | 1 +
27 files changed, 938 insertions(+), 48 deletions(-)
copy
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/exceptions/UnsupportedTxnActionException.java
=>
pulsar-common/src/main/java/org/apache/pulsar/common/intercept/AppendBrokerTimestampMetadataInterceptor.java
(63%)
copy
pulsar-common/src/main/java/org/apache/pulsar/common/intercept/{InterceptException.java
=> BrokerEntryMetadataInterceptor.java} (70%)
create mode 100644
pulsar-common/src/main/java/org/apache/pulsar/common/intercept/BrokerEntryMetadataUtils.java
copy tests/{bc_2_0_1 => bc_2_6_0}/pom.xml (87%)
copy tests/{bc_2_0_1 =>
bc_2_6_0}/src/test/java/org/apache/pulsar/tests/integration/PulsarContainer.java
(100%)
copy tests/{bc_2_0_1 =>
bc_2_6_0}/src/test/java/org/apache/pulsar/tests/integration/SmokeTest.java
(100%)
copy tests/{bc_2_0_1/src/test/resources/pulsar.xml =>
bc_2_6_0/src/test/resources/backwards-client.xml} (100%)
[pulsar] branch master updated (cdbbf4a -> 59660e4)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.
from cdbbf4a [docs] Configure authParams as JSON (#8870)
add 59660e4 fix authParams showing in log with secret string(*)
(#8910)
No new revisions were added by this update.
Summary of changes:
.../client/impl/conf/ClientConfigurationData.java | 5 ++-
.../client/util/{ConsumerName.java => Secret.java} | 17 +---
.../{MathUtils.java => SecretsSerializer.java} | 27 ++--
.../impl/conf/ConfigurationDataUtilsTest.java | 50 +-
4 files changed, 75 insertions(+), 24 deletions(-)
copy
pulsar-client/src/main/java/org/apache/pulsar/client/util/{ConsumerName.java =>
Secret.java} (62%)
copy pulsar-client/src/main/java/org/apache/pulsar/client/util/{MathUtils.java
=> SecretsSerializer.java} (59%)
[pulsar] branch master updated (91e2f83 -> 7f62bbc)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.
from 91e2f83 [docs] Fix incorrect command (#8758)
add 7f62bbc Split pulsar-client-and-admin-shade-test into two parts
(#8716)
No new revisions were added by this update.
Summary of changes:
tests/bc_2_0_0/pom.xml | 6 --
tests/bc_2_0_1/pom.xml | 6 --
tests/pom.xml| 3 ++-
.../pom.xml | 20 ++--
.../pulsar/tests/integration/PulsarContainer.java| 0
.../apache/pulsar/tests/integration/SmokeTest.java | 0
.../src/test/resources/pulsar.xml| 2 +-
tests/pulsar-client-all-shade-test/pom.xml | 6 --
.../pom.xml | 13 -
.../pulsar/tests/integration/PulsarContainer.java| 0
.../apache/pulsar/tests/integration/SmokeTest.java | 2 +-
.../src/test/resources/pulsar.xml| 2 +-
12 files changed, 11 insertions(+), 49 deletions(-)
rename tests/{pulsar-client-and-admin-shade-test =>
pulsar-client-admin-shade-test}/pom.xml (89%)
copy tests/{pulsar-client-and-admin-shade-test =>
pulsar-client-admin-shade-test}/src/test/java/org/apache/pulsar/tests/integration/PulsarContainer.java
(100%)
rename tests/{pulsar-client-and-admin-shade-test =>
pulsar-client-admin-shade-test}/src/test/java/org/apache/pulsar/tests/integration/SmokeTest.java
(100%)
rename tests/{pulsar-client-and-admin-shade-test =>
pulsar-client-admin-shade-test}/src/test/resources/pulsar.xml (94%)
copy tests/{pulsar-client-all-shade-test => pulsar-client-shade-test}/pom.xml
(92%)
rename tests/{pulsar-client-and-admin-shade-test =>
pulsar-client-shade-test}/src/test/java/org/apache/pulsar/tests/integration/PulsarContainer.java
(100%)
copy tests/{bc_2_0_0 =>
pulsar-client-shade-test}/src/test/java/org/apache/pulsar/tests/integration/SmokeTest.java
(97%)
copy tests/{pulsar-client-all-shade-test =>
pulsar-client-shade-test}/src/test/resources/pulsar.xml (94%)
[pulsar] branch master updated (b5b5c89 -> 8b15792)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.
from b5b5c89 Adopt switch tab for pulsar-admin, REST API, and Java (#8629)
add 8b15792 Introduce the packages management interface API (#8669)
No new revisions were added by this update.
Summary of changes:
pom.xml| 8 ++
.../core}/pom.xml | 27 ++--
.../management/core/PackagesManagement.java| 109 ++
.../packages/management/core/PackagesStorage.java | 82 +++
.../core/PackagesStorageConfiguration.java | 25 ++--
.../management/core/PackagesStorageProvider.java | 45 ++
.../management/core/common/PackageMetadata.java| 19 ++-
.../management/core/common/PackageName.java| 156 +
.../management/core/common/PackageType.java| 20 +--
.../management/core/common}/package-info.java | 4 +-
.../packages/management/core}/package-info.java| 9 +-
.../management/core/common/PackageNameTest.java| 119
.../pom.xml| 31 ++--
13 files changed, 555 insertions(+), 99 deletions(-)
copy {pulsar-io/common => pulsar-package-management/core}/pom.xml (69%)
create mode 100644
pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/PackagesManagement.java
create mode 100644
pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/PackagesStorage.java
copy
pulsar-broker-common/src/main/java/org/apache/pulsar/common/configuration/PropertyContext.java
=>
pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/PackagesStorageConfiguration.java
(62%)
copy
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/TransactionBufferProvider.java
=>
pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/PackagesStorageProvider.java
(51%)
copy
pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/DelayedDeliveryPolicies.java
=>
pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/common/PackageMetadata.java
(76%)
create mode 100644
pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/common/PackageName.java
copy
pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicDomain.java =>
pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/common/PackageType.java
(68%)
copy {pulsar-broker/src/main/java/org/apache/pulsar/broker/intercept =>
pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core/common}/package-info.java
(89%)
copy {pulsar-broker/src/main/java/org/apache/pulsar/broker/intercept =>
pulsar-package-management/core/src/main/java/org/apache/pulsar/packages/management/core}/package-info.java
(63%)
create mode 100644
pulsar-package-management/core/src/test/java/org/apache/pulsar/packages/management/core/common/PackageNameTest.java
copy {pulsar-client-api => pulsar-package-management}/pom.xml (78%)
[pulsar] branch branch-2.6 updated: Fix cherry-pick issue of the #8304 (#8639)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch branch-2.6
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.6 by this push:
new 3795d1f Fix cherry-pick issue of the #8304 (#8639)
3795d1f is described below
commit 3795d1f5f5c99361b79191eb422ff86f6e247e2d
Author: lipenghui
AuthorDate: Fri Nov 20 12:00:25 2020 +0800
Fix cherry-pick issue of the #8304 (#8639)
Fix cherry-pick issue of the #8304
---
.../src/main/java/org/apache/pulsar/zookeeper/ZooKeeperCache.java| 1 -
1 file changed, 1 deletion(-)
diff --git
a/pulsar-zookeeper-utils/src/main/java/org/apache/pulsar/zookeeper/ZooKeeperCache.java
b/pulsar-zookeeper-utils/src/main/java/org/apache/pulsar/zookeeper/ZooKeeperCache.java
index 26eb058..34b55a6 100644
---
a/pulsar-zookeeper-utils/src/main/java/org/apache/pulsar/zookeeper/ZooKeeperCache.java
+++
b/pulsar-zookeeper-utils/src/main/java/org/apache/pulsar/zookeeper/ZooKeeperCache.java
@@ -109,7 +109,6 @@ public abstract class ZooKeeperCache implements Watcher {
this.dataCache = Caffeine.newBuilder()
.recordStats()
-.expireAfterWrite(zkOperationTimeoutSeconds, TimeUnit.SECONDS)
.buildAsync((key, executor1) -> null);
this.childrenCache = Caffeine.newBuilder()
[pulsar] branch branch-2.6 updated: [bot-cherry-pick][broker] Close topics that remain fenced forcefully (#8636)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch branch-2.6
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.6 by this push:
new 291ec28 [bot-cherry-pick][broker] Close topics that remain fenced
forcefully (#8636)
291ec28 is described below
commit 291ec2836c3f5c20abeb722f3414e1720bafac7f
Author: lipenghui
AuthorDate: Fri Nov 20 11:59:42 2020 +0800
[bot-cherry-pick][broker] Close topics that remain fenced forcefully (#8636)
* [bot-cherry-pick][broker] Close topics that remain fenced forcefully
* Fix conflicts.
Co-authored-by: Masahiro Sakamoto
---
conf/broker.conf | 4 +++
conf/standalone.conf | 4 +++
.../apache/pulsar/broker/ServiceConfiguration.java | 6
.../broker/service/persistent/PersistentTopic.java | 41 --
.../pulsar/broker/service/PersistentTopicTest.java | 38
5 files changed, 91 insertions(+), 2 deletions(-)
diff --git a/conf/broker.conf b/conf/broker.conf
index 2709a83..444180e 100644
--- a/conf/broker.conf
+++ b/conf/broker.conf
@@ -410,6 +410,10 @@ systemTopicEnabled=false
# Please enable the system topic first.
topicLevelPoliciesEnabled=false
+# If a topic remains fenced for this number of seconds, it will be closed
forcefully.
+# If it is set to 0 or a negative number, the fenced topic will not be closed.
+topicFencingTimeoutSeconds=0
+
### --- Authentication --- ###
# Role names that are treated as "proxy roles". If the broker sees a request
with
#role as proxyRoles - it will demand to see a valid original principal.
diff --git a/conf/standalone.conf b/conf/standalone.conf
index 0ce88dc..7ca742a 100644
--- a/conf/standalone.conf
+++ b/conf/standalone.conf
@@ -350,6 +350,10 @@ systemTopicEnabled=false
# Please enable the system topic first.
topicLevelPoliciesEnabled=false
+# If a topic remains fenced for this number of seconds, it will be closed
forcefully.
+# If it is set to 0 or a negative number, the fenced topic will not be closed.
+topicFencingTimeoutSeconds=0
+
### --- Authentication --- ###
# Role names that are treated as "proxy roles". If the broker sees a request
with
#role as proxyRoles - it will demand to see a valid original principal.
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index 4da87d8..7d52f78 100644
---
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -755,6 +755,12 @@ public class ServiceConfiguration implements
PulsarConfiguration {
)
private String zookeeperSessionExpiredPolicy = "shutdown";
+@FieldContext(
+category = CATEGORY_SERVER,
+doc = "If a topic remains fenced for this number of seconds, it will
be closed forcefully.\n"
++ " If it is set to 0 or a negative number, the fenced topic
will not be closed."
+)
+private int topicFencingTimeoutSeconds = 0;
/ --- Messaging Protocols --- /
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index f467732..198195d 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -37,6 +37,7 @@ import java.util.Set;
import java.util.concurrent.CancellationException;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionException;
+import java.util.concurrent.ScheduledFuture;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicLong;
@@ -184,6 +185,8 @@ public class PersistentTopic extends AbstractTopic
implements Topic, AddEntryCal
public volatile int maxUnackedMessagesOnSubscription = -1;
private volatile boolean isClosingOrDeleting = false;
+private ScheduledFuture fencedTopicMonitoringTask = null;
+
private static class TopicStatsHelper {
public double averageMsgSize;
public double aggMsgRateIn;
@@ -353,7 +356,7 @@ public class PersistentTopic extends AbstractTopic
implements Topic, AddEntryCal
// signal to managed ledger that we are ready to resume by
creating a new ledger
ledger.readyToCreateNewLedger();
-isFenced = false;
+unfence();
}
}
@@ -380,7 +383,7 @@
[pulsar] branch master updated (48e98e4 -> c4da921)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 48e98e4 Remove CGO client from repo (#8514) add c4da921 Reduce memory leaks in pulsar-broker tests (#8526) No new revisions were added by this update. Summary of changes: .../pulsar/broker/service/BrokerService.java | 3 +- .../apache/pulsar/broker/SLAMonitoringTest.java| 2 +- .../broker/admin/AdminApiDelayedDelivery.java | 2 +- .../broker/admin/AdminApiGetLastMessageIdTest.java | 2 +- .../broker/admin/AdminApiMaxUnackedMessages.java | 2 +- .../pulsar/broker/admin/AdminApiOffloadTest.java | 12 ++- .../broker/admin/AdminApiSchemaAutoUpdateTest.java | 2 +- .../pulsar/broker/admin/AdminApiSchemaTest.java| 2 +- .../admin/AdminApiSchemaValidationEnforced.java| 2 +- .../apache/pulsar/broker/admin/AdminApiTest.java | 3 +- .../apache/pulsar/broker/admin/AdminApiTest2.java | 7 +- .../pulsar/broker/admin/AdminApiTlsAuthTest.java | 2 +- .../pulsar/broker/admin/AdminResourceTest.java | 2 +- .../org/apache/pulsar/broker/admin/AdminTest.java | 2 +- .../pulsar/broker/admin/AdminTopicApiTest.java | 2 +- .../apache/pulsar/broker/admin/BookiesApiTest.java | 2 +- .../broker/admin/BrokerAdminClientTlsAuthTest.java | 2 +- .../broker/admin/CreateSubscriptionTest.java | 2 +- .../broker/admin/IncrementPartitionsTest.java | 2 +- .../broker/admin/MaxUnackedMessagesTest.java | 2 +- .../apache/pulsar/broker/admin/NamespacesTest.java | 2 +- .../pulsar/broker/admin/PersistentTopicsTest.java | 4 +- .../pulsar/broker/admin/TopicAutoCreationTest.java | 2 +- .../pulsar/broker/admin/TopicMessageTTLTest.java | 3 +- .../broker/admin/TopicPoliciesDisableTest.java | 2 +- .../pulsar/broker/admin/TopicPoliciesTest.java | 2 +- .../pulsar/broker/admin/v1/V1_AdminApiTest.java| 2 +- .../pulsar/broker/admin/v1/V1_AdminApiTest2.java | 8 +- .../pulsar/broker/auth/AuthorizationTest.java | 2 +- .../broker/auth/MockedPulsarServiceBaseTest.java | 88 +++--- .../broker/cache/ResourceQuotaCacheTest.java | 2 +- .../broker/intercept/BrokerInterceptorTest.java| 2 +- .../AntiAffinityNamespaceGroupTest.java| 2 +- .../loadbalance/LeaderElectionServiceTest.java | 4 +- .../broker/loadbalance/LoadBalancerTest.java | 25 +++--- .../broker/loadbalance/LoadReportNetworkLimit.java | 2 +- .../loadbalance/ModularLoadManagerImplTest.java| 2 +- .../loadbalance/SimpleLoadManagerImplTest.java | 2 +- .../namespace/NamespaceCreateBundlesTest.java | 2 +- .../namespace/NamespaceOwnershipListenerTests.java | 2 +- .../broker/namespace/NamespaceServiceTest.java | 2 +- .../broker/namespace/NamespaceUnloadingTest.java | 2 +- .../OwnerShipCacheForCurrentServerTest.java| 2 +- .../broker/namespace/OwnershipCacheTest.java | 2 +- .../broker/protocol/ProtocolHandlersTest.java | 2 +- .../broker/service/AdvertisedAddressTest.java | 2 +- .../broker/service/BacklogQuotaManagerTest.java| 2 +- .../pulsar/broker/service/BatchMessageTest.java| 2 +- .../pulsar/broker/service/BkEnsemblesTestBase.java | 2 +- .../broker/service/BrokerBookieIsolationTest.java | 2 +- .../BrokerServiceAutoSubscriptionCreationTest.java | 4 +- .../BrokerServiceAutoTopicCreationTest.java| 4 +- .../pulsar/broker/service/BrokerServiceTest.java | 2 +- .../service/BrokerServiceThrottlingTest.java | 2 +- .../broker/service/DistributedIdGeneratorTest.java | 2 +- .../pulsar/broker/service/MaxMessageSizeTest.java | 2 +- .../broker/service/NonPersistentTopicE2ETest.java | 2 +- .../pulsar/broker/service/NullValueTest.java | 2 +- .../pulsar/broker/service/PartitionKeyTest.java| 2 +- .../pulsar/broker/service/PeerReplicatorTest.java | 2 +- .../PersistentDispatcherFailoverConsumerTest.java | 23 +- .../broker/service/PersistentFailoverE2ETest.java | 4 +- .../broker/service/PersistentQueueE2ETest.java | 2 +- .../broker/service/PersistentTopicE2ETest.java | 3 +- .../pulsar/broker/service/PersistentTopicTest.java | 4 +- .../pulsar/broker/service/RackAwareTest.java | 2 +- .../broker/service/ReplicatorGlobalNSTest.java | 2 +- .../broker/service/ReplicatorRateLimiterTest.java | 2 +- .../pulsar/broker/service/ReplicatorTest.java | 4 +- .../pulsar/broker/service/ReplicatorTestBase.java | 12 ++- .../pulsar/broker/service/ReplicatorTlsTest.java | 2 +- .../pulsar/broker/service/ResendRequestTest.java | 2 +- .../pulsar/broker/service/ServerCnxTest.java | 2 +- .../broker/service/SubscriptionSeekTest.java | 2 +- .../SystemTopicBasedTopicPoliciesServiceTest.java | 2 +- .../pulsar/broker/service/TopicOwnerTest.java | 2 +- .../service
[pulsar-presto] branch master updated: Add license header file and fix checkstyle (#2)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar-presto.git The following commit(s) were added to refs/heads/master by this push: new 46fd434 Add license header file and fix checkstyle (#2) 46fd434 is described below commit 46fd4341aebdd36d1efd22936c83e32aa9ac4d6e Author: Sijie Guo AuthorDate: Thu Nov 12 01:52:07 2020 -0700 Add license header file and fix checkstyle (#2) --- .github/workflows/unit-test.yaml | 2 +- presto-distribution/pom.xml | 9 - src/license-header.txt | 16 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/.github/workflows/unit-test.yaml b/.github/workflows/unit-test.yaml index 5927731..1447b34 100644 --- a/.github/workflows/unit-test.yaml +++ b/.github/workflows/unit-test.yaml @@ -84,7 +84,7 @@ jobs: - name: Build pulsar adapters run: | cd adapters - mvn clean license:check install + mvn clean license:check checkstyle:check install - name: package surefire artifacts if: failure() diff --git a/presto-distribution/pom.xml b/presto-distribution/pom.xml index f585ea0..2aee92e 100644 --- a/presto-distribution/pom.xml +++ b/presto-distribution/pom.xml @@ -339,7 +339,7 @@ license-maven-plugin 3.0 -../../src/license-header.txt +../src/license-header.txt @@ -349,6 +349,13 @@ 1.8 + + org.apache.maven.plugins + maven-checkstyle-plugin + + true + + diff --git a/src/license-header.txt b/src/license-header.txt new file mode 100644 index 000..60b675e --- /dev/null +++ b/src/license-header.txt @@ -0,0 +1,16 @@ +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License.
[pulsar-presto] branch master updated: Add license header file and fix checkstyle (#2)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar-presto.git The following commit(s) were added to refs/heads/master by this push: new 46fd434 Add license header file and fix checkstyle (#2) 46fd434 is described below commit 46fd4341aebdd36d1efd22936c83e32aa9ac4d6e Author: Sijie Guo AuthorDate: Thu Nov 12 01:52:07 2020 -0700 Add license header file and fix checkstyle (#2) --- .github/workflows/unit-test.yaml | 2 +- presto-distribution/pom.xml | 9 - src/license-header.txt | 16 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/.github/workflows/unit-test.yaml b/.github/workflows/unit-test.yaml index 5927731..1447b34 100644 --- a/.github/workflows/unit-test.yaml +++ b/.github/workflows/unit-test.yaml @@ -84,7 +84,7 @@ jobs: - name: Build pulsar adapters run: | cd adapters - mvn clean license:check install + mvn clean license:check checkstyle:check install - name: package surefire artifacts if: failure() diff --git a/presto-distribution/pom.xml b/presto-distribution/pom.xml index f585ea0..2aee92e 100644 --- a/presto-distribution/pom.xml +++ b/presto-distribution/pom.xml @@ -339,7 +339,7 @@ license-maven-plugin 3.0 -../../src/license-header.txt +../src/license-header.txt @@ -349,6 +349,13 @@ 1.8 + + org.apache.maven.plugins + maven-checkstyle-plugin + + true + + diff --git a/src/license-header.txt b/src/license-header.txt new file mode 100644 index 000..60b675e --- /dev/null +++ b/src/license-header.txt @@ -0,0 +1,16 @@ +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License.
[pulsar] branch master updated (76ff84a -> e3a9df7)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.
from 76ff84a Put 2.6.2 as a stable version (#8531)
add e3a9df7 Fix checkstyle violations (#8529)
No new revisions were added by this update.
Summary of changes:
.../apache/pulsar/broker/loadbalance/impl/SimpleLoadManagerImpl.java | 1 -
.../org/apache/pulsar/broker/service/persistent/PersistentTopic.java | 1 -
.../pulsar/broker/transaction/pendingack/PendingAckHandle.java | 3 +++
.../broker/transaction/pendingack/impl/PendingAckHandleImpl.java | 3 +++
.../{intercept => transaction/pendingack/impl}/package-info.java | 5 ++---
.../broker/{intercept => transaction/pendingack}/package-info.java | 5 ++---
6 files changed, 10 insertions(+), 8 deletions(-)
copy pulsar-broker/src/main/java/org/apache/pulsar/broker/{intercept =>
transaction/pendingack/impl}/package-info.java (86%)
copy pulsar-broker/src/main/java/org/apache/pulsar/broker/{intercept =>
transaction/pendingack}/package-info.java (87%)
[pulsar] branch master updated (c989661 -> 3d6377b)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from c989661 Support taking de-duplication snapshots based on time (#8474) add 3d6377b Update deps, allow to run tests with forkCount=1 (#8465) No new revisions were added by this update. Summary of changes: pom.xml| 12 +--- .../pulsar/broker/auth/MockedPulsarServiceBaseTest.java| 2 ++ .../java/org/apache/pulsar/admin/cli/CmdFunctionsTest.java | 14 +++--- .../pulsar/functions/utils/FunctionConfigUtilsTest.java| 2 +- tests/bc_2_0_0/src/test/resources/pulsar.xml | 2 +- tests/bc_2_0_1/src/test/resources/pulsar.xml | 2 +- .../src/test/resources/pulsar-backwards-compatibility.xml | 2 +- tests/integration/src/test/resources/pulsar-cli-suite.xml | 2 +- tests/integration/src/test/resources/pulsar-cli.xml| 2 +- .../src/test/resources/pulsar-function-state.xml | 2 +- tests/integration/src/test/resources/pulsar-messaging.xml | 2 +- .../src/test/resources/pulsar-process-suite.xml| 2 +- tests/integration/src/test/resources/pulsar-process.xml| 2 +- .../integration/src/test/resources/pulsar-schema-suite.xml | 2 +- tests/integration/src/test/resources/pulsar-schema.xml | 2 +- tests/integration/src/test/resources/pulsar-sql-suite.xml | 2 +- tests/integration/src/test/resources/pulsar-sql.xml| 2 +- tests/integration/src/test/resources/pulsar-standalone.xml | 2 +- .../integration/src/test/resources/pulsar-thread-suite.xml | 2 +- tests/integration/src/test/resources/pulsar-thread.xml | 2 +- tests/integration/src/test/resources/pulsar.xml| 2 +- .../src/test/resources/tiered-file-system-suite.xml| 2 +- .../src/test/resources/tiered-filesystem-storage.xml | 2 +- .../src/test/resources/tiered-jcloud-storage.xml | 2 +- .../src/test/resources/tiered-storage-jcloud-suite.xml | 2 +- 25 files changed, 40 insertions(+), 32 deletions(-)
[pulsar] branch master updated (c989661 -> 3d6377b)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from c989661 Support taking de-duplication snapshots based on time (#8474) add 3d6377b Update deps, allow to run tests with forkCount=1 (#8465) No new revisions were added by this update. Summary of changes: pom.xml| 12 +--- .../pulsar/broker/auth/MockedPulsarServiceBaseTest.java| 2 ++ .../java/org/apache/pulsar/admin/cli/CmdFunctionsTest.java | 14 +++--- .../pulsar/functions/utils/FunctionConfigUtilsTest.java| 2 +- tests/bc_2_0_0/src/test/resources/pulsar.xml | 2 +- tests/bc_2_0_1/src/test/resources/pulsar.xml | 2 +- .../src/test/resources/pulsar-backwards-compatibility.xml | 2 +- tests/integration/src/test/resources/pulsar-cli-suite.xml | 2 +- tests/integration/src/test/resources/pulsar-cli.xml| 2 +- .../src/test/resources/pulsar-function-state.xml | 2 +- tests/integration/src/test/resources/pulsar-messaging.xml | 2 +- .../src/test/resources/pulsar-process-suite.xml| 2 +- tests/integration/src/test/resources/pulsar-process.xml| 2 +- .../integration/src/test/resources/pulsar-schema-suite.xml | 2 +- tests/integration/src/test/resources/pulsar-schema.xml | 2 +- tests/integration/src/test/resources/pulsar-sql-suite.xml | 2 +- tests/integration/src/test/resources/pulsar-sql.xml| 2 +- tests/integration/src/test/resources/pulsar-standalone.xml | 2 +- .../integration/src/test/resources/pulsar-thread-suite.xml | 2 +- tests/integration/src/test/resources/pulsar-thread.xml | 2 +- tests/integration/src/test/resources/pulsar.xml| 2 +- .../src/test/resources/tiered-file-system-suite.xml| 2 +- .../src/test/resources/tiered-filesystem-storage.xml | 2 +- .../src/test/resources/tiered-jcloud-storage.xml | 2 +- .../src/test/resources/tiered-storage-jcloud-suite.xml | 2 +- 25 files changed, 40 insertions(+), 32 deletions(-)
[pulsar] branch master updated: Upgrade Apache BookKeeper Client to 4.11.1 (#8270)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 465964b Upgrade Apache BookKeeper Client to 4.11.1 (#8270) 465964b is described below commit 465964bee516921b7d2067256676942ae55a Author: Enrico Olivelli AuthorDate: Fri Oct 30 13:03:28 2020 +0100 Upgrade Apache BookKeeper Client to 4.11.1 (#8270) Motivation Upgrade Apache BookKeeper client to version 4.11.1 and RocksDB to 6.10.2 (needed by Bookie) Modifications Upgrade Apache BookKeeper client, fix a few build errors, due to the introduction of BookieServiceInfo structure (see BP-38). Verifying this change This change is already covered by existing tests * Upgrade Apache BookKeeper to 4.11.1 * fix build * fix LICENSE files * Use BK 4.11.1 * Upgrade RocksDB to 6.10.2 * Fix license and upgrade maven assembly plugin * update license * restore bouncy castle license * fix licenses * Exclude BouncyCastle inherited from BK Co-authored-by: Enrico Olivelli Co-authored-by: Enrico Olivelli --- distribution/io/pom.xml| 1 + distribution/offloaders/pom.xml| 1 + .../server/licenses/LICENSE-bouncycastle.txt | 20 ++-- distribution/server/pom.xml| 1 + distribution/server/src/assemble/LICENSE.bin.txt | 58 -- .../bookkeeper/test/BookKeeperClusterTestCase.java | 4 +- pom.xml| 12 - .../org/apache/pulsar/PulsarBrokerStarter.java | 3 +- .../pulsar/broker/service/RackAwareTest.java | 3 +- .../broker/transaction/TransactionTestBase.java| 1 - pulsar-functions/runtime-all/pom.xml | 1 + pulsar-sql/presto-distribution/LICENSE | 47 +- pulsar-sql/presto-distribution/pom.xml | 32 pulsar-sql/presto-pulsar-plugin/pom.xml| 1 + .../pulsar/zookeeper/LocalBookkeeperEnsemble.java | 9 ++-- src/check-binary-license | 2 +- 16 files changed, 118 insertions(+), 78 deletions(-) diff --git a/distribution/io/pom.xml b/distribution/io/pom.xml index 13b96d2..8e061adc 100644 --- a/distribution/io/pom.xml +++ b/distribution/io/pom.xml @@ -51,6 +51,7 @@ maven-assembly-plugin +3.3.0 distro-assembly diff --git a/distribution/offloaders/pom.xml b/distribution/offloaders/pom.xml index 1a12683..4aed9c1 100644 --- a/distribution/offloaders/pom.xml +++ b/distribution/offloaders/pom.xml @@ -45,6 +45,7 @@ maven-assembly-plugin +3.3.0 distro-assembly diff --git a/distribution/server/licenses/LICENSE-bouncycastle.txt b/distribution/server/licenses/LICENSE-bouncycastle.txt index 6f1dedf..10a3dd4 100644 --- a/distribution/server/licenses/LICENSE-bouncycastle.txt +++ b/distribution/server/licenses/LICENSE-bouncycastle.txt @@ -1,19 +1,7 @@ -The Bouncy Castle License +Copyright (c) 2000 - 2020 The Legion of the Bouncy Castle Inc. (https://www.bouncycastle.org) -Copyright (c) 2000-2015 The Legion Of The Bouncy Castle Inc. (http://www.bouncycastle.org) +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: -Permission is hereby granted, free of charge, to any person obtaining a copy of this software -and associated documentation files (the "Software"), to deal in the Software without restriction, -including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, -and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, -subject to the following conditions: +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. -The above copyright notice and this permission notice shall be included in all copies or substantial -portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, -INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR -PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE -LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
[pulsar] branch master updated (c5705f2 -> b527ccc)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from c5705f2 [Issue 8311][pulsar-client-go] Fix memory leak in cgo golang client (#8325) add b527ccc Allow building Apache Pulsar on JDK15+ - upgrade Maven Assembly Plugin (#8360) No new revisions were added by this update. Summary of changes: distribution/server/src/assemble/LICENSE.bin.txt | 1 - pom.xml| 35 ++ .../pulsar-client-kafka/pom.xml| 2 +- pulsar-common/pom.xml | 24 +++ pulsar-sql/presto-distribution/LICENSE | 34 - pulsar-sql/presto-distribution/pom.xml | 31 +-- src/check-binary-license | 2 +- .../kafka/PulsarKafkaProducerThreadSafeTest.java | 1 - 8 files changed, 83 insertions(+), 47 deletions(-)
[pulsar] branch master updated (8dda4d0 -> 5b93ae1)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 8dda4d0 [Issue 8364][docs] Update decommission content in different releases (#8368) add 5b93ae1 sync 2.6.0 docs into 2.6.1 and 2.6.2 (#8398) No new revisions were added by this update. Summary of changes: .../version-2.6.1/getting-started-helm.md | 51 +- .../version-2.6.2/getting-started-helm.md | 51 +- 2 files changed, 42 insertions(+), 60 deletions(-)
[pulsar] branch master updated: [Issue 8154] [Python client] Expose schema version (of writerSchema) in Message (#8173)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 54d6811 [Issue 8154] [Python client] Expose schema version (of
writerSchema) in Message (#8173)
54d6811 is described below
commit 54d6811bf163fae6cae569c77c2537e33012cfdc
Author: Shivji Kumar Jha
AuthorDate: Wed Oct 28 14:20:54 2020 +0530
[Issue 8154] [Python client] Expose schema version (of writerSchema) in
Message (#8173)
* [Issue 8154] Expose schema version (of writerSchema) in python client
message
* Adding formating suggestion on PR#8173 to fix tests
* Fixing build issues
* Added a test for python client returning schema version
* Added one more test case for python client returning schema version
* Fix test- move subscribe before send so the consumer offset is ahead of
new data
* Fix test to make it run on python 2 and 3 both
Co-authored-by: Sijie Guo
---
pulsar-client-cpp/include/pulsar/Message.h | 10 ++
pulsar-client-cpp/include/pulsar/c/message.h | 6 ++
pulsar-client-cpp/lib/ConsumerImpl.cc| 4
pulsar-client-cpp/lib/Message.cc | 14 ++
pulsar-client-cpp/lib/MessageImpl.cc | 6 ++
pulsar-client-cpp/lib/MessageImpl.h | 6 ++
pulsar-client-cpp/python/pulsar/__init__.py | 6 ++
pulsar-client-cpp/python/schema_test.py | 26 ++
pulsar-client-cpp/python/src/message.cc | 7 +++
9 files changed, 85 insertions(+)
diff --git a/pulsar-client-cpp/include/pulsar/Message.h
b/pulsar-client-cpp/include/pulsar/Message.h
index d1775a6..0a750e3 100644
--- a/pulsar-client-cpp/include/pulsar/Message.h
+++ b/pulsar-client-cpp/include/pulsar/Message.h
@@ -152,6 +152,16 @@ class PULSAR_PUBLIC Message {
*/
const int getRedeliveryCount() const;
+/**
+ * Check if schema version exists
+ */
+bool hasSchemaVersion() const;
+
+/**
+ * Get the schema version
+ */
+const std::string& getSchemaVersion() const;
+
bool operator==(const Message& msg) const;
private:
diff --git a/pulsar-client-cpp/include/pulsar/c/message.h
b/pulsar-client-cpp/include/pulsar/c/message.h
index dcb9152..f54d025 100644
--- a/pulsar-client-cpp/include/pulsar/c/message.h
+++ b/pulsar-client-cpp/include/pulsar/c/message.h
@@ -200,6 +200,12 @@ PULSAR_PUBLIC const char
*pulsar_message_get_topic_name(pulsar_message_t *messag
PULSAR_PUBLIC int pulsar_message_get_redelivery_count(pulsar_message_t
*message);
+PULSAR_PUBLIC int pulsar_message_has_schema_version(pulsar_message_t *message);
+
+PULSAR_PUBLIC const char *pulsar_message_get_schemaVersion(pulsar_message_t
*message);
+
+PULSAR_PUBLIC void pulsar_message_set_schema_version(pulsar_message_t
*message, const char *schemaVersion);
+
#ifdef __cplusplus
}
#endif
diff --git a/pulsar-client-cpp/lib/ConsumerImpl.cc
b/pulsar-client-cpp/lib/ConsumerImpl.cc
index b6a0c15..6f9cdbf 100644
--- a/pulsar-client-cpp/lib/ConsumerImpl.cc
+++ b/pulsar-client-cpp/lib/ConsumerImpl.cc
@@ -327,6 +327,10 @@ void ConsumerImpl::messageReceived(const
ClientConnectionPtr& cnx, const proto::
m.impl_->setTopicName(topic_);
m.impl_->setRedeliveryCount(msg.redelivery_count());
+if (metadata.has_schema_version()) {
+m.impl_->setSchemaVersion(metadata.schema_version());
+}
+
LOG_DEBUG(getName() << " metadata.num_messages_in_batch() = " <<
metadata.num_messages_in_batch());
LOG_DEBUG(getName() << " metadata.has_num_messages_in_batch() = "
<< metadata.has_num_messages_in_batch());
diff --git a/pulsar-client-cpp/lib/Message.cc b/pulsar-client-cpp/lib/Message.cc
index 5b135bc..76e408f 100644
--- a/pulsar-client-cpp/lib/Message.cc
+++ b/pulsar-client-cpp/lib/Message.cc
@@ -145,6 +145,20 @@ const int Message::getRedeliveryCount() const {
return impl_->getRedeliveryCount();
}
+bool Message::hasSchemaVersion() const {
+if (impl_) {
+return impl_->hasSchemaVersion();
+}
+return false;
+}
+
+const std::string& Message::getSchemaVersion() const {
+if (!impl_) {
+return emptyString;
+}
+return impl_->getSchemaVersion();
+}
+
uint64_t Message::getPublishTimestamp() const { return impl_ ?
impl_->getPublishTimestamp() : 0ull; }
uint64_t Message::getEventTimestamp() const { return impl_ ?
impl_->getEventTimestamp() : 0ull; }
diff --git a/pulsar-client-cpp/lib/MessageImpl.cc
b/pulsar-client-cpp/lib/MessageImpl.cc
index b41e7ae..5d1edbf 100644
--- a/pulsar-client-cpp/lib/MessageImpl.cc
+++ b/pulsar-client-cpp/lib/MessageImpl.cc
@@ -96,4 +96,10 @@ int MessageImpl::getRedeliveryCount() { return
redeliveryCount_; }
void MessageI
[pulsar] branch master updated (db5320d -> 9eead7d)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from db5320d [Issue 8346][Admin API] Validate retention policy (#8358) add 9eead7d Support partitioned topic lookup (#7605) No new revisions were added by this update. Summary of changes: .../pulsar/broker/lookup/v2/TopicLookup.java | 1 - .../apache/pulsar/broker/admin/AdminApiTest.java | 4 ++ .../pulsar/broker/service/TopicOwnerTest.java | 33 +++ .../org/apache/pulsar/client/admin/Lookup.java | 18 ++ .../apache/pulsar/client/admin/PulsarAdmin.java| 2 +- .../pulsar/client/admin/internal/LookupImpl.java | 66 +- .../org/apache/pulsar/admin/cli/CliCommand.java| 7 +++ .../org/apache/pulsar/admin/cli/CmdTopics.java | 13 + 8 files changed, 141 insertions(+), 3 deletions(-)
[pulsar] branch master updated (db5320d -> 9eead7d)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from db5320d [Issue 8346][Admin API] Validate retention policy (#8358) add 9eead7d Support partitioned topic lookup (#7605) No new revisions were added by this update. Summary of changes: .../pulsar/broker/lookup/v2/TopicLookup.java | 1 - .../apache/pulsar/broker/admin/AdminApiTest.java | 4 ++ .../pulsar/broker/service/TopicOwnerTest.java | 33 +++ .../org/apache/pulsar/client/admin/Lookup.java | 18 ++ .../apache/pulsar/client/admin/PulsarAdmin.java| 2 +- .../pulsar/client/admin/internal/LookupImpl.java | 66 +- .../org/apache/pulsar/admin/cli/CliCommand.java| 7 +++ .../org/apache/pulsar/admin/cli/CmdTopics.java | 13 + 8 files changed, 141 insertions(+), 3 deletions(-)
[pulsar-test-infra] branch master updated: Detect when there are changes in the root directory (#11)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-test-infra.git
The following commit(s) were added to refs/heads/master by this push:
new ca1cd67 Detect when there are changes in the root directory (#11)
ca1cd67 is described below
commit ca1cd67593aeda83b0ca5b48a261b70f4bd90c58
Author: Lari Hotari
AuthorDate: Mon Oct 26 04:53:56 2020 +0200
Detect when there are changes in the root directory (#11)
Fixes #10
The problem is that `git diff --dirstat=files,0 ` doesn't
show changes in the root directory.
Replace with
`git diff --name-only | awk -F "/*[^/]*/*$" '{ print ($1 == "" ?
"." : $1); }' | sort | uniq`
which will list changes in the root directory as "."
---
diff-only/entrypoint.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/diff-only/entrypoint.sh b/diff-only/entrypoint.sh
index d2c6f86..1b5f5d1 100755
--- a/diff-only/entrypoint.sh
+++ b/diff-only/entrypoint.sh
@@ -9,7 +9,7 @@ echo "COMMITS: ${COMMITS}"
git --version
git rev-parse --abbrev-ref HEAD
-CHANGED_DIRS=$(git diff --dirstat=files,0 HEAD~${COMMITS} | awk '{ print $2 }')
+CHANGED_DIRS=$(git diff --name-only HEAD~${COMMITS} | awk -F "/*[^/]*/*$" '{
print ($1 == "" ? "." : $1); }' | sort | uniq)
echo "CHANGED_DIRS are : ${CHANGED_DIRS}"
found_changed_dir_not_in_target_dirs="no"
[pulsar] branch master updated: [C++] Fix message id error when subscribing a single partition (#8341)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new dd8be83 [C++] Fix message id error when subscribing a single
partition (#8341)
dd8be83 is described below
commit dd8be8323d04d252062575840f5f6be7487796b5
Author: Yunze Xu
AuthorDate: Fri Oct 23 22:02:33 2020 +0800
[C++] Fix message id error when subscribing a single partition (#8341)
### Motivation
A consumer/reader can subscribe a single partition of a partitioned topic.
However, the partition index was not set correctly. So the partition field of
received messages' id is always -1.
In addition, current `TopicName#getPartitionIndex` Java implementation
doesn't handle some corner cases, like "xxx-partition-00" or
"xxx-partition--1". For the backward compatibility, the C++ implementation will
use the same implementation and note it in Java `TopicName`'s unit test.
### Modifications
- Add `getPartitionIndex` method to `TopicName`;
- Set the partition index when a `Consumer` was created from a `Client`;
- Set the partition index when a `Reader` was created;
- Add relative unit tests for above changes;
- Note the "incorrect" behavior in Java `TopicName`'s unit test.
### Verifying this change
This change added tests and can be verified as follows:
- `TopicNameTest#testPartitionIndex`
- `ConsumerTest#testPartitionIndex`
- `ReaderTest#testPartitionIndex`
* Add getPartitionIndex() method to TopicName
* Set partition index when creating a ConsumerImpl from Client
* Add test cases to Java TopicName to note the corner cases
* Set partition index for Reader
* Fix compile error
---
pulsar-client-cpp/lib/ClientImpl.cc| 4 +-
pulsar-client-cpp/lib/ReaderImpl.cc| 2 +
pulsar-client-cpp/lib/TopicName.cc | 18 ++
pulsar-client-cpp/lib/TopicName.h | 3 +
pulsar-client-cpp/tests/ConsumerTest.cc| 65 ++
pulsar-client-cpp/tests/ReaderTest.cc | 44 +++
pulsar-client-cpp/tests/TopicNameTest.cc | 25 +
.../apache/pulsar/common/naming/TopicNameTest.java | 5 ++
8 files changed, 165 insertions(+), 1 deletion(-)
diff --git a/pulsar-client-cpp/lib/ClientImpl.cc
b/pulsar-client-cpp/lib/ClientImpl.cc
index f0ce634..b46ec35 100644
--- a/pulsar-client-cpp/lib/ClientImpl.cc
+++ b/pulsar-client-cpp/lib/ClientImpl.cc
@@ -368,8 +368,10 @@ void ClientImpl::handleSubscribe(const Result result,
const LookupDataResultPtr
consumer =
std::make_shared(shared_from_this(), consumerName,
topicName,
partitionMetadata->getPartitions(), conf);
} else {
-consumer =
+auto consumerImpl =
std::make_shared(shared_from_this(),
topicName->toString(), consumerName, conf);
+consumerImpl->setPartitionIndex(topicName->getPartitionIndex());
+consumer = consumerImpl;
}
consumer->getConsumerCreatedFuture().addListener(
std::bind(::handleConsumerCreated, shared_from_this(),
std::placeholders::_1,
diff --git a/pulsar-client-cpp/lib/ReaderImpl.cc
b/pulsar-client-cpp/lib/ReaderImpl.cc
index 4bd091a..1e08977 100644
--- a/pulsar-client-cpp/lib/ReaderImpl.cc
+++ b/pulsar-client-cpp/lib/ReaderImpl.cc
@@ -19,6 +19,7 @@
#include "ClientImpl.h"
#include "ReaderImpl.h"
+#include "TopicName.h"
namespace pulsar {
@@ -54,6 +55,7 @@ void ReaderImpl::start(const MessageId& startMessageId) {
consumer_ = std::make_shared(
client_.lock(), topic_, subscription, consumerConf,
ExecutorServicePtr(), NonPartitioned,
Commands::SubscriptionModeNonDurable,
Optional::of(startMessageId));
+consumer_->setPartitionIndex(TopicName::getPartitionIndex(topic_));
consumer_->getConsumerCreatedFuture().addListener(std::bind(::handleConsumerCreated,
shared_from_this(), std::placeholders::_1,
std::placeholders::_2));
diff --git a/pulsar-client-cpp/lib/TopicName.cc
b/pulsar-client-cpp/lib/TopicName.cc
index 3a5e281..46fa355 100644
--- a/pulsar-client-cpp/lib/TopicName.cc
+++ b/pulsar-client-cpp/lib/TopicName.cc
@@ -90,6 +90,7 @@ bool TopicName::init(const std::string& topicName) {
} else {
namespaceName_ = NamespaceName::get(property_, cluster_,
namespacePortion_);
}
+partition_ = TopicName::getPartitionIndex(localName_);
return true;
}
bool TopicName::parse(const std::string& t
[pulsar] branch master updated (ef614ad -> ca97be3)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.
from ef614ad Remove bouncy castle shaded module to avoid bring error of
verifySingleJar (#7453)
add ca97be3 Add proxy plugin interface to support user defined additional
servlet (#8067)
No new revisions were added by this update.
Summary of changes:
.../broker/intercept/BrokerInterceptorUtils.java | 2 +-
.../pulsar/proxy/server/ProxyConfiguration.java| 21 +++
.../apache/pulsar/proxy/server/ProxyService.java | 13 +-
.../pulsar/proxy/server/ProxyServiceStarter.java | 14 ++
.../org/apache/pulsar/proxy/server/WebServer.java | 3 +-
.../plugin/servlet/ProxyAdditionalServlet.java | 33 ++--
.../servlet/ProxyAdditionalServletDefinition.java | 15 +-
.../servlet/ProxyAdditionalServletDefinitions.java | 9 +-
.../servlet/ProxyAdditionalServletMetadata.java| 13 +-
.../servlet/ProxyAdditionalServletUtils.java | 160
.../ProxyAdditionalServletWithClassLoader.java | 47 ++---
.../plugin/servlet/ProxyAdditionalServlets.java| 89 +
.../proxy/server/plugin/servlet}/package-info.java | 4 +-
.../proxy/server/ProxyAdditionalServletTest.java | 201 +
.../server/plugin/MockProxyAdditionalServlet.java | 28 +--
.../plugin/ProxyAdditionalServletUtilsTest.java| 77
.../ProxyAdditionalServletWithClassLoaderTest.java | 29 ++-
17 files changed, 619 insertions(+), 139 deletions(-)
copy
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/SchemaInfoProvider.java
=>
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/plugin/servlet/ProxyAdditionalServlet.java
(54%)
copy
pulsar-broker/src/main/java/org/apache/pulsar/broker/intercept/BrokerInterceptorDefinition.java
=>
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/plugin/servlet/ProxyAdditionalServletDefinition.java
(72%)
copy
pulsar-broker/src/main/java/org/apache/pulsar/broker/protocol/ProtocolHandlerDefinitions.java
=>
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/plugin/servlet/ProxyAdditionalServletDefinitions.java
(79%)
copy
pulsar-broker/src/main/java/org/apache/pulsar/broker/protocol/ProtocolHandlerMetadata.java
=>
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/plugin/servlet/ProxyAdditionalServletMetadata.java
(76%)
create mode 100644
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/plugin/servlet/ProxyAdditionalServletUtils.java
copy
pulsar-broker/src/main/java/org/apache/pulsar/broker/protocol/ProtocolHandlerWithClassLoader.java
=>
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/plugin/servlet/ProxyAdditionalServletWithClassLoader.java
(53%)
create mode 100644
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/plugin/servlet/ProxyAdditionalServlets.java
copy {pulsar-broker/src/main/java/org/apache/pulsar/broker/intercept =>
pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/plugin/servlet}/package-info.java
(90%)
create mode 100644
pulsar-proxy/src/test/java/org/apache/pulsar/proxy/server/ProxyAdditionalServletTest.java
copy
pulsar-io/batch-data-generator/src/main/java/org/apache/pulsar/io/batchdatagenerator/BatchDataGeneratorPrintSink.java
=>
pulsar-proxy/src/test/java/org/apache/pulsar/proxy/server/plugin/MockProxyAdditionalServlet.java
(62%)
copy
pulsar-broker/src/test/java/org/apache/pulsar/broker/intercept/BrokerInterceptorUtilsTest.java
=>
pulsar-proxy/src/test/java/org/apache/pulsar/proxy/server/plugin/ProxyAdditionalServletUtilsTest.java
(58%)
copy
pulsar-broker/src/test/java/org/apache/pulsar/broker/intercept/BrokerInterceptorWithClassLoaderTest.java
=>
pulsar-proxy/src/test/java/org/apache/pulsar/proxy/server/plugin/ProxyAdditionalServletWithClassLoaderTest.java
(56%)
[pulsar] branch master updated (7f9b7cf -> ef614ad)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 7f9b7cf [pulsar-broker] configure namespace anti-affinity in local policies (#8349) add ef614ad Remove bouncy castle shaded module to avoid bring error of verifySingleJar (#7453) No new revisions were added by this update. Summary of changes: bouncy-castle/bc-shaded/pom.xml| 84 - bouncy-castle/bc/pom.xml | 18 ++- bouncy-castle/bcfips-include-test/pom.xml | 4 +- .../pulsar/client/TlsProducerConsumerTest.java | 4 + bouncy-castle/bcfips-nar-test/pom.xml | 102 .../pulsar/client/TlsProducerConsumerBase.java | 130 - .../pulsar/client/TlsProducerConsumerTest.java | 130 - .../resources/authentication/tls/broker-cert.pem | 73 .../resources/authentication/tls/broker-key.pem| 28 - .../test/resources/authentication/tls/cacert.pem | 62 -- .../resources/authentication/tls/client-cert.pem | 73 .../resources/authentication/tls/client-key.pem| 28 - bouncy-castle/bcfips/pom.xml | 17 ++- bouncy-castle/pom.xml | 4 - distribution/server/pom.xml| 10 +- distribution/server/src/assemble/LICENSE.bin.txt | 6 + pulsar-broker-shaded/pom.xml | 7 ++ pulsar-broker/pom.xml | 15 +-- .../client/api/SimpleProducerConsumerTest.java | 34 -- pulsar-client-all/pom.xml | 7 ++ .../pulsar-client-kafka-tests/pom.xml | 8 ++ .../pulsar-client-kafka-tests_0_8/pom.xml | 8 ++ .../pulsar-client-kafka-tests_0_9/pom.xml | 8 ++ .../pulsar-client-kafka_0_8/pom.xml| 7 ++ pulsar-client-messagecrypto-bc/pom.xml | 4 +- pulsar-client-shaded/pom.xml | 7 ++ pulsar-client-tools-test/pom.xml | 1 + pulsar-client/pom.xml | 5 +- .../apache/pulsar/common/util/BcNarDefinition.java | 47 .../pulsar/common/util/SearchBcNarUtils.java | 118 --- .../apache/pulsar/common/util/SecurityUtility.java | 9 +- pulsar-discovery-service/pom.xml | 11 +- pulsar-functions/worker/pom.xml| 22 ++-- pulsar-io/debezium/core/pom.xml| 7 ++ pulsar-io/kafka-connect-adaptor/pom.xml| 7 ++ pulsar-sql/presto-distribution/LICENSE | 5 +- pulsar-sql/presto-pulsar-plugin/pom.xml| 9 +- pulsar-sql/presto-pulsar/pom.xml | 8 ++ pulsar-testclient/pom.xml | 9 ++ pulsar-websocket/pom.xml | 8 ++ site2/docs/security-bouncy-castle.md | 1 + .../version-2.5.1/security-bouncy-castle.md| 1 + .../version-2.5.2/security-bouncy-castle.md| 1 + .../version-2.6.0/security-bouncy-castle.md| 1 + .../version-2.6.1/security-bouncy-castle.md| 1 + tests/integration/pom.xml | 7 ++ tests/pulsar-kafka-compat-client-test/pom.xml | 7 ++ tests/pulsar-storm-test/pom.xml| 10 +- 48 files changed, 236 insertions(+), 937 deletions(-) delete mode 100644 bouncy-castle/bc-shaded/pom.xml delete mode 100644 bouncy-castle/bcfips-nar-test/pom.xml delete mode 100644 bouncy-castle/bcfips-nar-test/src/test/java/org/apache/pulsar/client/TlsProducerConsumerBase.java delete mode 100644 bouncy-castle/bcfips-nar-test/src/test/java/org/apache/pulsar/client/TlsProducerConsumerTest.java delete mode 100644 bouncy-castle/bcfips-nar-test/src/test/resources/authentication/tls/broker-cert.pem delete mode 100644 bouncy-castle/bcfips-nar-test/src/test/resources/authentication/tls/broker-key.pem delete mode 100644 bouncy-castle/bcfips-nar-test/src/test/resources/authentication/tls/cacert.pem delete mode 100644 bouncy-castle/bcfips-nar-test/src/test/resources/authentication/tls/client-cert.pem delete mode 100644 bouncy-castle/bcfips-nar-test/src/test/resources/authentication/tls/client-key.pem delete mode 100644 pulsar-common/src/main/java/org/apache/pulsar/common/util/BcNarDefinition.java delete mode 100644 pulsar-common/src/main/java/org/apache/pulsar/common/util/SearchBcNarUtils.java
[pulsar] branch master updated (7f9b7cf -> ef614ad)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 7f9b7cf [pulsar-broker] configure namespace anti-affinity in local policies (#8349) add ef614ad Remove bouncy castle shaded module to avoid bring error of verifySingleJar (#7453) No new revisions were added by this update. Summary of changes: bouncy-castle/bc-shaded/pom.xml| 84 - bouncy-castle/bc/pom.xml | 18 ++- bouncy-castle/bcfips-include-test/pom.xml | 4 +- .../pulsar/client/TlsProducerConsumerTest.java | 4 + bouncy-castle/bcfips-nar-test/pom.xml | 102 .../pulsar/client/TlsProducerConsumerBase.java | 130 - .../pulsar/client/TlsProducerConsumerTest.java | 130 - .../resources/authentication/tls/broker-cert.pem | 73 .../resources/authentication/tls/broker-key.pem| 28 - .../test/resources/authentication/tls/cacert.pem | 62 -- .../resources/authentication/tls/client-cert.pem | 73 .../resources/authentication/tls/client-key.pem| 28 - bouncy-castle/bcfips/pom.xml | 17 ++- bouncy-castle/pom.xml | 4 - distribution/server/pom.xml| 10 +- distribution/server/src/assemble/LICENSE.bin.txt | 6 + pulsar-broker-shaded/pom.xml | 7 ++ pulsar-broker/pom.xml | 15 +-- .../client/api/SimpleProducerConsumerTest.java | 34 -- pulsar-client-all/pom.xml | 7 ++ .../pulsar-client-kafka-tests/pom.xml | 8 ++ .../pulsar-client-kafka-tests_0_8/pom.xml | 8 ++ .../pulsar-client-kafka-tests_0_9/pom.xml | 8 ++ .../pulsar-client-kafka_0_8/pom.xml| 7 ++ pulsar-client-messagecrypto-bc/pom.xml | 4 +- pulsar-client-shaded/pom.xml | 7 ++ pulsar-client-tools-test/pom.xml | 1 + pulsar-client/pom.xml | 5 +- .../apache/pulsar/common/util/BcNarDefinition.java | 47 .../pulsar/common/util/SearchBcNarUtils.java | 118 --- .../apache/pulsar/common/util/SecurityUtility.java | 9 +- pulsar-discovery-service/pom.xml | 11 +- pulsar-functions/worker/pom.xml| 22 ++-- pulsar-io/debezium/core/pom.xml| 7 ++ pulsar-io/kafka-connect-adaptor/pom.xml| 7 ++ pulsar-sql/presto-distribution/LICENSE | 5 +- pulsar-sql/presto-pulsar-plugin/pom.xml| 9 +- pulsar-sql/presto-pulsar/pom.xml | 8 ++ pulsar-testclient/pom.xml | 9 ++ pulsar-websocket/pom.xml | 8 ++ site2/docs/security-bouncy-castle.md | 1 + .../version-2.5.1/security-bouncy-castle.md| 1 + .../version-2.5.2/security-bouncy-castle.md| 1 + .../version-2.6.0/security-bouncy-castle.md| 1 + .../version-2.6.1/security-bouncy-castle.md| 1 + tests/integration/pom.xml | 7 ++ tests/pulsar-kafka-compat-client-test/pom.xml | 7 ++ tests/pulsar-storm-test/pom.xml| 10 +- 48 files changed, 236 insertions(+), 937 deletions(-) delete mode 100644 bouncy-castle/bc-shaded/pom.xml delete mode 100644 bouncy-castle/bcfips-nar-test/pom.xml delete mode 100644 bouncy-castle/bcfips-nar-test/src/test/java/org/apache/pulsar/client/TlsProducerConsumerBase.java delete mode 100644 bouncy-castle/bcfips-nar-test/src/test/java/org/apache/pulsar/client/TlsProducerConsumerTest.java delete mode 100644 bouncy-castle/bcfips-nar-test/src/test/resources/authentication/tls/broker-cert.pem delete mode 100644 bouncy-castle/bcfips-nar-test/src/test/resources/authentication/tls/broker-key.pem delete mode 100644 bouncy-castle/bcfips-nar-test/src/test/resources/authentication/tls/cacert.pem delete mode 100644 bouncy-castle/bcfips-nar-test/src/test/resources/authentication/tls/client-cert.pem delete mode 100644 bouncy-castle/bcfips-nar-test/src/test/resources/authentication/tls/client-key.pem delete mode 100644 pulsar-common/src/main/java/org/apache/pulsar/common/util/BcNarDefinition.java delete mode 100644 pulsar-common/src/main/java/org/apache/pulsar/common/util/SearchBcNarUtils.java
[pulsar] branch master updated (80e2fec -> 3147e9e)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 80e2fec #7994 Missed dryRun on maintenance of secrets. (#8286) add 3147e9e [Issue 8130] Refine BookKeeper metadata service uri when init metadata (#8269) No new revisions were added by this update. Summary of changes: .../java/org/apache/pulsar/PulsarClusterMetadataSetup.java| 8 .../pulsar/broker/zookeeper/ClusterMetadataSetupTest.java | 2 +- site2/docs/deploy-bare-metal.md | 11 +++ site2/docs/reference-cli-tools.md | 7 ++- 4 files changed, 22 insertions(+), 6 deletions(-)
[pulsar] branch master updated (d3a5233 -> 5df23b5)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from d3a5233 Refactored JCloud Tiered Storage (#6335) add 5df23b5 Fix Broker enters an infinite loop in ManagedLedgerImpl.asyncReadEntries (#8284) No new revisions were added by this update. Summary of changes: .../java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java| 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[pulsar] branch branch-2.6 updated: Don't fail the health check request when try to delete previous subscription. (#7906)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch branch-2.6
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.6 by this push:
new 49b8c6f Don't fail the health check request when try to delete
previous subscription. (#7906)
49b8c6f is described below
commit 49b8c6f1d7b127dac4e87e3c01e8d4c9f36533ae
Author: lipenghui
AuthorDate: Tue Sep 1 13:18:23 2020 +0800
Don't fail the health check request when try to delete previous
subscription. (#7906)
(cherry picked from commit 800681aaa9af6b588c482aa93f3303f1dd6ca668)
---
.../java/org/apache/pulsar/broker/admin/impl/BrokersBase.java | 11 ---
1 file changed, 8 insertions(+), 3 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
index e78427c..192c59a 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
@@ -267,11 +267,16 @@ public class BrokersBase extends AdminResource {
// create non-partitioned topic manually and close the previous reader
if present.
try {
pulsar().getBrokerService().getTopic(topic,
true).get().ifPresent(t -> {
-
t.getSubscriptions().values().forEach(Subscription::deleteForcefully);
+for (Subscription value : t.getSubscriptions().values()) {
+try {
+value.deleteForcefully();
+} catch (Exception e) {
+LOG.warn("Failed to delete previous subscription {}
for health check", value.getName(), e);
+}
+}
});
} catch (Exception e) {
-asyncResponse.resume(new RestException(e));
-return;
+LOG.warn("Failed to try to delete subscriptions for health check",
e);
}
CompletableFuture> producerFuture =
client.newProducer(Schema.STRING).topic(topic).createAsync();
[pulsar] 02/02: Set dryrun of KubernetesRuntime is null (#8064)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch branch-2.6
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 3b8ce21c5ec7b64b17c357088d82c268a0003477
Author: xiaolong ran
AuthorDate: Fri Sep 18 10:45:04 2020 +0800
Set dryrun of KubernetesRuntime is null (#8064)
Signed-off-by: xiaolong.ran
### Motivation
In #7994, we upgrade the `client-java` of Kubernetes to `0.9.2`, this will
cause the following error. During the creation of statefulsets, secrets and
services, the code has been configured to set the "dryrun" value of "true".
"true" is not accepted by Kubernetes. Only "All" is allowed. So the value
should be set to null.
### Modifications
- Set the `dryrun` of KubernetesRuntime is null.
(cherry picked from commit ad8abd5d2605f621c32a316f7a415f1ba00df130)
---
.../pulsar/functions/runtime/kubernetes/KubernetesRuntime.java| 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java
b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java
index aff638c..16bfd74 100644
---
a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java
+++
b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java
@@ -419,7 +419,7 @@ public class KubernetesRuntime implements Runtime {
.supplier(() -> {
final V1Service response;
try {
-response =
coreClient.createNamespacedService(jobNamespace, service, null, "true", null);
+response =
coreClient.createNamespacedService(jobNamespace, service, null, null, null);
} catch (ApiException e) {
// already exists
if (e.getCode() == HTTP_CONFLICT) {
@@ -504,7 +504,7 @@ public class KubernetesRuntime implements Runtime {
.supplier(() -> {
final V1StatefulSet response;
try {
-response =
appsClient.createNamespacedStatefulSet(jobNamespace, statefulSet, null, "true",
null);
+response =
appsClient.createNamespacedStatefulSet(jobNamespace, statefulSet, null, null,
null);
} catch (ApiException e) {
// already exists
if (e.getCode() == HTTP_CONFLICT) {
@@ -555,7 +555,7 @@ public class KubernetesRuntime implements Runtime {
// https://github.com/kubernetes-client/java/issues/86
response = appsClient.deleteNamespacedStatefulSetCall(
statefulSetName,
-jobNamespace, null, "true",
+jobNamespace, null, null,
5, null, "Foreground",
null, null)
.execute();
@@ -706,7 +706,7 @@ public class KubernetesRuntime implements Runtime {
// https://github.com/kubernetes-client/java/issues/86
response = coreClient.deleteNamespacedServiceCall(
serviceName,
-jobNamespace, null, "true",
+jobNamespace, null, null,
0, null,
"Foreground", null, null).execute();
} catch (ApiException e) {
[pulsar] branch branch-2.6 updated (3e65a6c -> 3b8ce21)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch branch-2.6 in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 3e65a6c Replace com.ning.asynchttpclient with org.asynchttpclient (#8099) new 9d02404 [issue 7851][C++] Make clear() thread-safe (#7862) new 3b8ce21 Set dryrun of KubernetesRuntime is null (#8064) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc | 1 + pulsar-client-cpp/lib/UnAckedMessageTrackerEnabled.cc | 6 +- pulsar-client-cpp/tests/BasicEndToEndTest.cc | 2 ++ .../pulsar/functions/runtime/kubernetes/KubernetesRuntime.java| 8 4 files changed, 12 insertions(+), 5 deletions(-)
[pulsar] 01/02: [issue 7851][C++] Make clear() thread-safe (#7862)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch branch-2.6
in repository https://gitbox.apache.org/repos/asf/pulsar.git
commit 9d024040499805a0536d745b2f9ddce3da9ca983
Author: Yunze Xu
AuthorDate: Mon Aug 24 07:39:32 2020 +0800
[issue 7851][C++] Make clear() thread-safe (#7862)
Fixes #7851
### Motivation
`clear()` methods of `BatchAcknowledgementTracker` and
`UnAckedMessageTrackerEnabled` are not thread-safe.
### Modifications
Acquire a mutex in these `clear()` methods.
(cherry picked from commit 97f41120b9691474f0038b220f3204fa69e48257)
---
pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc | 1 +
pulsar-client-cpp/lib/UnAckedMessageTrackerEnabled.cc | 6 +-
pulsar-client-cpp/tests/BasicEndToEndTest.cc | 2 ++
3 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc
b/pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc
index df15119..3d6d920 100644
--- a/pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc
+++ b/pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc
@@ -33,6 +33,7 @@
BatchAcknowledgementTracker::BatchAcknowledgementTracker(const std::string topic
}
void BatchAcknowledgementTracker::clear() {
+Lock lock(mutex_);
trackerMap_.clear();
sendList_.clear();
}
diff --git a/pulsar-client-cpp/lib/UnAckedMessageTrackerEnabled.cc
b/pulsar-client-cpp/lib/UnAckedMessageTrackerEnabled.cc
index 9185dba..e280dba 100644
--- a/pulsar-client-cpp/lib/UnAckedMessageTrackerEnabled.cc
+++ b/pulsar-client-cpp/lib/UnAckedMessageTrackerEnabled.cc
@@ -39,7 +39,7 @@ void UnAckedMessageTrackerEnabled::timeoutHandler() {
}
void UnAckedMessageTrackerEnabled::timeoutHandlerHelper() {
-std::lock_guard acquire(lock_);
+std::unique_lock acquire(lock_);
LOG_DEBUG("UnAckedMessageTrackerEnabled::timeoutHandlerHelper invoked for
consumerPtr_ "
<< consumerReference_.getName().c_str());
@@ -60,6 +60,9 @@ void UnAckedMessageTrackerEnabled::timeoutHandlerHelper() {
timePartitions.push_back(headPartition);
if (msgIdsToRedeliver.size() > 0) {
+// redeliverUnacknowledgedMessages() may call clear() that acquire the
lock again, so we should unlock
+// here to avoid deadlock
+acquire.unlock();
consumerReference_.redeliverUnacknowledgedMessages(msgIdsToRedeliver);
}
}
@@ -148,6 +151,7 @@ void UnAckedMessageTrackerEnabled::removeTopicMessage(const
std::string& topic)
}
void UnAckedMessageTrackerEnabled::clear() {
+std::lock_guard acquire(lock_);
messageIdPartitionMap.clear();
for (auto it = timePartitions.begin(); it != timePartitions.end(); it++) {
it->clear();
diff --git a/pulsar-client-cpp/tests/BasicEndToEndTest.cc
b/pulsar-client-cpp/tests/BasicEndToEndTest.cc
index cc71a35..acad16a 100644
--- a/pulsar-client-cpp/tests/BasicEndToEndTest.cc
+++ b/pulsar-client-cpp/tests/BasicEndToEndTest.cc
@@ -1748,6 +1748,8 @@ TEST(BasicEndToEndTest,
testPartitionTopicUnAckedMessageTimeout) {
std::this_thread::sleep_for(std::chrono::milliseconds(500));
timeWaited += 500;
}
+
+client.close();
}
TEST(BasicEndToEndTest, testUnAckedMessageTimeoutListener) {
[pulsar] branch master updated: Add read position when joining in the consumer stats. (#8274)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 8a846dc Add read position when joining in the consumer stats. (#8274)
8a846dc is described below
commit 8a846dc24624c2c7fee2ea91c2f8fafb9fddabf6
Author: lipenghui
AuthorDate: Fri Oct 16 20:49:16 2020 +0800
Add read position when joining in the consumer stats. (#8274)
Motivation
To troubleshooting the consumer stop to receive messages under the
key_shared subscription, it's difficult to know if the broker stops to dispatch
messages to this consumer to preserve order. So this PR exposes a metrics to
show the read position when the consumer joining so that we can compare the
read position and the mark delete position to determine.
Verifying this change
Unit test added
---
.../org/apache/pulsar/broker/service/Consumer.java | 8 +
...istentStickyKeyDispatcherMultipleConsumers.java | 9 +++---
.../apache/pulsar/broker/admin/AdminApiTest.java | 37 ++
.../pulsar/common/policies/data/ConsumerStats.java | 8 +
4 files changed, 58 insertions(+), 4 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
index 0e93d35..3a5625b 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
@@ -131,6 +131,7 @@ public class Consumer {
private static final double avgPercent = 0.9;
private boolean preciseDispatcherFlowControl;
+private PositionImpl readPositionWhenJoining;
public Consumer(Subscription subscription, SubType subType, String
topicName, long consumerId,
int priorityLevel, String consumerName,
@@ -553,6 +554,9 @@ public class Consumer {
stats.unackedMessages = unackedMessages;
stats.blockedConsumerOnUnackedMsgs = blockedConsumerOnUnackedMsgs;
stats.avgMessagesPerEntry = getAvgMessagesPerEntry();
+if (readPositionWhenJoining != null) {
+stats.readPositionWhenJoining = readPositionWhenJoining.toString();
+}
return stats;
}
@@ -739,5 +743,9 @@ public class Consumer {
return preciseDispatcherFlowControl;
}
+public void setReadPositionWhenJoining(PositionImpl
readPositionWhenJoining) {
+this.readPositionWhenJoining = readPositionWhenJoining;
+}
+
private static final Logger log = LoggerFactory.getLogger(Consumer.class);
}
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java
index 3cde9a1..1585cae 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java
@@ -107,12 +107,14 @@ public class
PersistentStickyKeyDispatcherMultipleConsumers extends PersistentDi
throw e;
}
+PositionImpl readPositionWhenJoining = (PositionImpl)
cursor.getReadPosition();
+consumer.setReadPositionWhenJoining(readPositionWhenJoining);
// If this was the 1st consumer, or if all the messages are already
acked, then we
// don't need to do anything special
-if (allowOutOfOrderDelivery == false
+if (!allowOutOfOrderDelivery
&& consumerList.size() > 1
&& cursor.getNumberOfEntriesSinceFirstNotAckedMessage() > 1) {
-recentlyJoinedConsumers.put(consumer, (PositionImpl)
cursor.getReadPosition());
+recentlyJoinedConsumers.put(consumer, readPositionWhenJoining);
}
}
@@ -154,8 +156,7 @@ public class PersistentStickyKeyDispatcherMultipleConsumers
extends PersistentDi
final Map> groupedEntries =
localGroupedEntries.get();
groupedEntries.clear();
-for (int i = 0; i < entriesCount; i++) {
-Entry entry = entries.get(i);
+for (Entry entry : entries) {
Consumer c = selector.select(peekStickyKey(entry.getDataBuffer()));
groupedEntries.computeIfAbsent(c, k -> new
ArrayList<>()).add(entry);
}
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest.java
index e8c92fd..e2bd1d9 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest.java
+++
b/p
[pulsar] branch master updated: Fix stuck lookup operations when the broker is starting up (#8273)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new b57c163 Fix stuck lookup operations when the broker is starting up
(#8273)
b57c163 is described below
commit b57c1630e2478755c05a147bfaf11d9a723cd28e
Author: Matteo Merli
AuthorDate: Fri Oct 16 05:49:02 2020 -0700
Fix stuck lookup operations when the broker is starting up (#8273)
Motivation
When the broker is starting up, it might start getting lookup requests
before all the components of the service are fully initialized. In this
particular case a lookup will fail on NPE because the leader election service
is not ready yet (it gets instantiated after the broker service).
This NPE causes a series of rippling effects:
The future for the request hitting NPE are not completed
They stay stale in the findingBundlesNotAuthoritative cache map forever
All other lookup requests are piggy-backing on the first futures (but these
will not complete)
We reach the max number of pending lookup requests, after which the broker
rejects new lookup
---
.../apache/pulsar/broker/namespace/NamespaceService.java| 13 -
1 file changed, 12 insertions(+), 1 deletion(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
index c00e802..511ea12 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
@@ -30,6 +30,7 @@ import org.apache.pulsar.broker.PulsarServerException;
import org.apache.pulsar.broker.PulsarService;
import org.apache.pulsar.broker.ServiceConfiguration;
import org.apache.pulsar.broker.admin.AdminResource;
+import org.apache.pulsar.broker.loadbalance.LeaderElectionService;
import org.apache.pulsar.broker.loadbalance.LoadManager;
import org.apache.pulsar.broker.loadbalance.ResourceUnit;
import org.apache.pulsar.broker.lookup.LookupResult;
@@ -404,7 +405,17 @@ public class NamespaceService {
return;
}
String candidateBroker = null;
-boolean authoritativeRedirect =
pulsar.getLeaderElectionService().isLeader();
+
+LeaderElectionService les = pulsar.getLeaderElectionService();
+if (les == null) {
+// The leader election service was not initialized yet. This can
happen because the broker service is
+// initialized first and it might start receiving lookup requests
before the leader election service is
+// fully initialized.
+lookupFuture.complete(Optional.empty());
+return;
+}
+
+boolean authoritativeRedirect = les.isLeader();
try {
// check if this is Heartbeat or SLAMonitor namespace
[pulsar] branch branch-2.6 updated: [C++] Fix some pending requests may never complete when broker's down (#8232)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch branch-2.6
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.6 by this push:
new bf83b1b [C++] Fix some pending requests may never complete when
broker's down (#8232)
bf83b1b is described below
commit bf83b1b29dcb77a740b27d9ab2369fdf78b1d4b4
Author: Yunze Xu
AuthorDate: Mon Oct 12 15:14:39 2020 +0800
[C++] Fix some pending requests may never complete when broker's down
(#8232)
Fixes #8230
### Motivation
When `ClientConnection` calls `close`, some promises of pending request
were not completed. It may cause some methods that wait for promise become
completed never return, like `Reader::hasMessageAvailable`.
### Modifications
- Complete `GetLastMessageId` and `GetNamespaceTopics` requests in
`ClientConnection::close`.
- Refactor `ClientConnection::close`, use move instead of swap to process
some fields outside the lock section.
- Add error log before connection was closed by read error.
### Verifying this change
Try to reproduce #8230 , `Reader::hasMessageAvailable` would return
`ResultConnectError` if broker was killed by force.
(cherry picked from commit 5e60775fcf607be202641a45166c5d8781e3de87)
---
pulsar-client-cpp/lib/ClientConnection.cc | 72 +++
1 file changed, 35 insertions(+), 37 deletions(-)
diff --git a/pulsar-client-cpp/lib/ClientConnection.cc
b/pulsar-client-cpp/lib/ClientConnection.cc
index ebb7268..6394d25 100644
--- a/pulsar-client-cpp/lib/ClientConnection.cc
+++ b/pulsar-client-cpp/lib/ClientConnection.cc
@@ -498,6 +498,9 @@ void ClientConnection::handleRead(const
boost::system::error_code& err, size_t b
incomingBuffer_.bytesWritten(bytesTransferred);
if (err || bytesTransferred == 0) {
+if (err) {
+LOG_ERROR(cnxString_ << "Read failed: " << err.message());
+} // else: bytesTransferred == 0, which means server has closed the
connection
close();
} else if (bytesTransferred < minReadSize) {
// Read the remaining part, use a slice of buffer to write on the next
@@ -1378,28 +1381,39 @@ void ClientConnection::close() {
state_ = Disconnected;
boost::system::error_code err;
socket_->close(err);
-ConsumersMap consumers;
-consumers.swap(consumers_);
-ProducersMap producers;
-producers.swap(producers_);
-lock.unlock();
-LOG_INFO(cnxString_ << "Connection closed");
+if (tlsSocket_) {
+tlsSocket_->lowest_layer().close();
+}
+
+if (executor_) {
+executor_.reset();
+}
+
+// Move the internal fields to process them after `mutex_` was unlocked
+auto consumers = std::move(consumers_);
+auto producers = std::move(producers_);
+auto pendingRequests = std::move(pendingRequests_);
+auto pendingLookupRequests = std::move(pendingLookupRequests_);
+auto pendingConsumerStatsMap = std::move(pendingConsumerStatsMap_);
+auto pendingGetLastMessageIdRequests =
std::move(pendingGetLastMessageIdRequests_);
+auto pendingGetNamespaceTopicsRequests =
std::move(pendingGetNamespaceTopicsRequests_);
+
+numOfPendingLookupRequest_ = 0;
if (keepAliveTimer_) {
-lock.lock();
keepAliveTimer_->cancel();
keepAliveTimer_.reset();
-lock.unlock();
}
if (consumerStatsRequestTimer_) {
-lock.lock();
consumerStatsRequestTimer_->cancel();
consumerStatsRequestTimer_.reset();
-lock.unlock();
}
+lock.unlock();
+LOG_INFO(cnxString_ << "Connection closed");
+
for (ProducersMap::iterator it = producers.begin(); it != producers.end();
++it) {
HandlerBase::handleDisconnection(ResultConnectError,
shared_from_this(), it->second);
}
@@ -1410,38 +1424,22 @@ void ClientConnection::close() {
connectPromise_.setFailed(ResultConnectError);
-// Fail all pending operations on the connection
-for (PendingRequestsMap::iterator it = pendingRequests_.begin(); it !=
pendingRequests_.end(); ++it) {
-it->second.promise.setFailed(ResultConnectError);
+// Fail all pending requests, all these type are map whose value type
contains the Promise object
+for (auto& kv : pendingRequests) {
+kv.second.promise.setFailed(ResultConnectError);
}
-
-// Fail all pending lookup-requests on the connection
-lock.lock();
-PendingLookupRequestsMap pendingLookupRequests;
-pendingLookupRequests_.swap(pendingLookupRequests);
-numOfPendingLookupRequest_ -= pendingLookupRequests.size();
-
-PendingConsumerStatsMap pendingConsumerStatsMap;
-pendingConsumerStatsMap_.swap(pendingConsumerStatsMap);
-lock.unlock
[pulsar] branch master updated (c6550e4 -> 5e60775)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from c6550e4 Add log level config in pulsar-client (#8195) add 5e60775 [C++] Fix some pending requests may never complete when broker's down (#8232) No new revisions were added by this update. Summary of changes: pulsar-client-cpp/lib/ClientConnection.cc | 72 +++ 1 file changed, 35 insertions(+), 37 deletions(-)
[pulsar] branch master updated (4c489be -> d70d821)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 4c489be Fix issue where paused consumer receives new message when reconnecting (#8165) add d70d821 fix examples/spark compile failed under jdk10 (#8194) No new revisions were added by this update. Summary of changes: examples/spark/pom.xml | 5 + pom.xml| 1 + 2 files changed, 6 insertions(+)
[pulsar] branch master updated: Add command to delete a cluster's metadata from ZK (#8169)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new d41729f Add command to delete a cluster's metadata from ZK (#8169)
d41729f is described below
commit d41729f422b215f53bd88c2eec6d1d112e804778
Author: Yunze Xu
AuthorDate: Thu Oct 1 12:24:40 2020 +0800
Add command to delete a cluster's metadata from ZK (#8169)
Motivation
When we share the same ZK and BK cluster with multiple broker clusters, if
a cluster was removed, its metadata in ZK should also be removed.
Modifications
Add a PulsarClusterMetadataTeardown class to delete the relative nodes from
ZK;
Wrap the class to bin/pulsar script.
---
bin/pulsar | 3 +
.../pulsar/PulsarClusterMetadataTeardown.java | 111 +
2 files changed, 114 insertions(+)
diff --git a/bin/pulsar b/bin/pulsar
index 1fe5332..0c20c8d 100755
--- a/bin/pulsar
+++ b/bin/pulsar
@@ -143,6 +143,7 @@ where command is one of:
standalone Run a broker server with local bookies and local
zookeeper
initialize-cluster-metadata One-time metadata initialization
+delete-cluster-metadata Delete a cluster's metadata
initialize-transaction-coordinator-metadata One-time transaction
coordinator metadata initialization
initialize-namespace namespace initialization
compact-topic Run compaction against a topic
@@ -339,6 +340,8 @@ elif [ $COMMAND == "standalone" ]; then
exec $JAVA $OPTS $ASPECTJ_AGENT ${ZK_OPTS}
-Dpulsar.log.file=$PULSAR_LOG_FILE org.apache.pulsar.PulsarStandaloneStarter
--config $PULSAR_STANDALONE_CONF $@
elif [ $COMMAND == "initialize-cluster-metadata" ]; then
exec $JAVA $OPTS org.apache.pulsar.PulsarClusterMetadataSetup $@
+elif [ $COMMAND == "delete-cluster-metadata" ]; then
+exec $JAVA $OPTS org.apache.pulsar.PulsarClusterMetadataTeardown $@
elif [ $COMMAND == "initialize-transaction-coordinator-metadata" ]; then
exec $JAVA $OPTS
org.apache.pulsar.PulsarTransactionCoordinatorMetadataSetup $@
elif [ $COMMAND == "initialize-namespace" ]; then
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataTeardown.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataTeardown.java
new file mode 100644
index 000..a9a1c11
--- /dev/null
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarClusterMetadataTeardown.java
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.pulsar.zookeeper.ZooKeeperClientFactory;
+import org.apache.pulsar.zookeeper.ZookeeperClientFactoryImpl;
+import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.ZKUtil;
+import org.apache.zookeeper.ZooKeeper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.concurrent.ExecutionException;
+
+/**
+ * Teardown the metadata for a existed Pulsar cluster
+ */
+public class PulsarClusterMetadataTeardown {
+
+private static class Arguments {
+@Parameter(names = { "-zk",
+"--zookeeper"}, description = "Local ZooKeeper quorum
connection string", required = true)
+private String zookeeper;
+
+@Parameter(names = {
+"--zookeeper-session-timeout-ms"
+}, description = "Local zookeeper session timeout ms")
+private int zkSessionTimeoutMillis = 3;
+
+@Parameter(names = { "-c", "-cluster" }, description = "Cluster name")
+private String cluster;
+
+@Parameter(names = { "-cs", "--configuration-store" }, description =
"Configuration Store connection string")
+private String configurationStore;
+
+@Parameter(names = { "-h", "--help"
[pulsar] branch master updated: [pulsar-admin] Support delete schema ledgers when delete topics (#8167)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 983266d [pulsar-admin] Support delete schema ledgers when delete
topics (#8167)
983266d is described below
commit 983266d480f77543a29a74ac1970280abd9f804b
Author: Yunze Xu
AuthorDate: Wed Sep 30 22:15:49 2020 +0800
[pulsar-admin] Support delete schema ledgers when delete topics (#8167)
Master Issue: #8134
### Motivation
The REST APIs of `PersistentTopics#deleteTopic` and
`PersistentTopics#deletePartitionedTopic` didn't support delete schema of
topics, which may cause after topics were deleted, the schema ledgers still
existed. And current implementation of delete schema is just add a empty schema
ledger but not delete existed schema ledgers.
### Modifications
- Add a `deleteSchema` query param to REST APIs of deleting
topics/partitioned topics;
- Add a map to record the created ledgers in `BookkeeperSchemaStorage`;
- Expose `deleteSchema` param in pulsar-admin APIs;
- Delete schema ledgers when delete the cluster with `-a` option.
### Verifying this change
It has been tested in local environments:
1. Check the current count of ledgers;
2. Create a new partitioned topic or non-partitioned topic;
3. Create producers/consumers with schema to send and receive messages with
the created topic;
4. Delete the topic with `deleteSchema=true` query param or call admin API
that set `deleteSchema` to false;
5. Recheck the count of ledgers, make sure it's the same with step 1.
---
.../broker/admin/impl/PersistentTopicsBase.java| 29 ++
.../pulsar/broker/admin/v1/PersistentTopics.java | 2 +-
.../pulsar/broker/admin/v2/PersistentTopics.java | 12 --
.../pulsar/broker/service/AbstractTopic.java | 6 ++-
.../pulsar/broker/service/BrokerService.java | 32 +++-
.../service/schema/BookkeeperSchemaStorage.java| 38 ++-
.../schema/DefaultSchemaRegistryService.java | 5 +++
.../broker/service/schema/SchemaRegistry.java | 2 +
.../service/schema/SchemaRegistryServiceImpl.java | 5 +++
...hemaRegistryServiceWithSchemaDataValidator.java | 5 +++
.../pulsar/broker/admin/PersistentTopicsTest.java | 4 +-
.../org/apache/pulsar/client/admin/Topics.java | 44 --
.../pulsar/client/admin/internal/TopicsImpl.java | 22 ++-
.../org/apache/pulsar/admin/cli/CmdClusters.java | 11 +++---
14 files changed, 179 insertions(+), 38 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
index 13c592e..18b0f62 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java
@@ -374,11 +374,11 @@ public class PersistentTopicsBase extends AdminResource {
grantPermissions(topicName.toString(), role, actions);
}
-protected void internalDeleteTopicForcefully(boolean authoritative) {
+protected void internalDeleteTopicForcefully(boolean authoritative,
boolean deleteSchema) {
validateWriteOperationOnTopic(authoritative);
try {
-pulsar().getBrokerService().deleteTopic(topicName.toString(),
true).get();
+pulsar().getBrokerService().deleteTopic(topicName.toString(),
true, deleteSchema).get();
} catch (Exception e) {
if (e.getCause() instanceof MetadataNotFoundException) {
log.info("[{}] Topic was already not existing {}",
clientAppId(), topicName, e);
@@ -630,7 +630,8 @@ public class PersistentTopicsBase extends AdminResource {
return metadata;
}
-protected void internalDeletePartitionedTopic(AsyncResponse asyncResponse,
boolean authoritative, boolean force) {
+protected void internalDeletePartitionedTopic(AsyncResponse asyncResponse,
boolean authoritative,
+ boolean force, boolean
deleteSchema) {
try {
validateWriteOperationOnTopic(authoritative);
} catch (WebApplicationException wae) {
@@ -650,6 +651,18 @@ public class PersistentTopicsBase extends AdminResource {
final int numPartitions = partitionMeta.partitions;
if (numPartitions > 0) {
final AtomicInteger count = new AtomicInteger(numPartitions);
+if (deleteSchema) {
+count.incrementAndGet();
+
pulsar().getBrokerService().deleteSchemaStorage(topicName.getPartition(0).toString())
+.whenCo
[pulsar] branch master updated (692a28e -> 170f12c)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 692a28e Fix JavaDoc issues. Spotted by IntelliJ (#8054) add 170f12c [Issue 8138][pulsar-client] Fix ConsumerImpl memory leaks (#8160) No new revisions were added by this update. Summary of changes: .../apache/pulsar/client/impl/ConsumerImpl.java| 57 ++ 1 file changed, 38 insertions(+), 19 deletions(-)
[pulsar] branch master updated: [pulsar-admin-tools] Support delete all data associated with a cluster (#8133)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 0b041a2 [pulsar-admin-tools] Support delete all data associated with
a cluster (#8133)
0b041a2 is described below
commit 0b041a2af8bdc0340d9d41b301b1ccc973056800
Author: Yunze Xu
AuthorDate: Tue Sep 29 08:50:17 2020 +0800
[pulsar-admin-tools] Support delete all data associated with a cluster
(#8133)
### Motivation
When multiple broker clusters shared the same bookie cluster, if user
wanted to remove a broker cluster, the associated ledgers in bookies should
also be deleted.
### Modifications
- Add an option to `cluster delete` command to delete all the data
associated with the cluster.
Currently there's no way to delete the ledgers of schema from client's
side, a new REST API to delete schema's ledgers should be exposed or the
existed `SchemaRegistryServiceImpl#deleteSchema`'s semantics should be change.
### Verifying this change
The new option has been tested in local environment by:
1. Deploy a ZK which acts as both local ZK and the configuration store;
2. Init metadata of 2 clusters, the 2nd cluster use
`--bookkeeper-metadata-service` option to share the same BK metadata position;
3. Deploy a BK;
4. Deploy 2 brokers associated with the 2 clusters' metadata;
5. Create some topics and produce some messages to 2 clusters;
6. Run `bin/pulsar cluster -all ` to delete the whole cluster, use
`bin/bookkeeper shell list ledgers` to verify the change of ledgers count.
**An integration test is needed to simulate the procedures above.**
---
.../java/org/apache/pulsar/admin/cli/CmdClusters.java | 19 +++
1 file changed, 19 insertions(+)
diff --git
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdClusters.java
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdClusters.java
index 8ea2e36..a489398 100644
---
a/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdClusters.java
+++
b/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdClusters.java
@@ -119,8 +119,27 @@ public class CmdClusters extends CmdBase {
@Parameter(description = "cluster-name\n", required = true)
private java.util.List params;
+@Parameter(names = { "-a", "--all" }, description = "Delete all data
(tenants) of the cluster\n", required = false)
+private boolean deleteAll = false;
+
void run() throws PulsarAdminException {
String cluster = getOneArgument(params);
+
+if (deleteAll) {
+for (String tenant : admin.tenants().getTenants()) {
+for (String namespace :
admin.namespaces().getNamespaces(tenant)) {
+for (String topic : admin.topics().getList(namespace))
{
+admin.topics().delete(topic, true);
+// TODO: Delete all the ledgers of the
SchemaStorage
+// admin.schemas().deleteSchema(topic) won't
delete the schema's ledger. Instead a new ledger will be created.
+//
https://github.com/apache/pulsar/issues/8134
+}
+admin.namespaces().deleteNamespace(namespace);
+}
+admin.tenants().deleteTenant(tenant);
+}
+}
+
admin.clusters().deleteCluster(cluster);
}
}
[pulsar] branch master updated: [Issue 8138][pulsar-client] Improve timeout handling in ClientCnx to cover all remaining request types (GetLastMessageId, GetTopics, GetSchema, GetOrCreateSchema) (#814
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new bf6a88e [Issue 8138][pulsar-client] Improve timeout handling in
ClientCnx to cover all remaining request types (GetLastMessageId, GetTopics,
GetSchema, GetOrCreateSchema) (#8149)
bf6a88e is described below
commit bf6a88e864cb6f23d4cadc64cb3790182d68a814
Author: Lari Hotari
AuthorDate: Tue Sep 29 03:34:26 2020 +0300
[Issue 8138][pulsar-client] Improve timeout handling in ClientCnx to cover
all remaining request types (GetLastMessageId, GetTopics, GetSchema,
GetOrCreateSchema) (#8149)
* Add timeout handling for all other request types in ClientCnx
Master Issue: #8138
### Motivation
Since one of the root causes for #8138 is that there isn't timeout handling
for GetLastMessageId requests, this PR adds that.
The PR improves timeout handling in ClientCnx to cover all remaining
request types that don't currently have timeout handling. The request types are:
- GetLastMessageId
- GetTopics
- GetSchema
- GetOrCreateSchema
### Modifications
The existing requestTimeoutQueue solution to handle "ordinary" command
requests has been extended to cover all the requests types.
### Verifying this change
This change added tests and can be verified as follows:
- Added unit tests in a new test class ClientCnxRequestTimeoutQueueTest to
test request timeout handling.
- `cd pulsar-client; mvn test -Dtest=ClientCnxRequestTimeoutQueueTest`
---
.../org/apache/pulsar/client/impl/ClientCnx.java | 118 +
.../client/impl/TransactionMetaStoreHandler.java | 11 +-
.../impl/ClientCnxRequestTimeoutQueueTest.java | 106 ++
3 files changed, 165 insertions(+), 70 deletions(-)
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
index 6f8c511..936b27f 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
@@ -119,7 +119,7 @@ public class ClientCnx extends PulsarHandler {
private final ConcurrentLongHashMap
transactionMetaStoreHandlers = new ConcurrentLongHashMap<>(16, 1);
private final CompletableFuture connectionFuture = new
CompletableFuture();
-private final ConcurrentLinkedQueue requestTimeoutQueue = new
ConcurrentLinkedQueue<>();
+private final ConcurrentLinkedQueue> requestTimeoutQueue =
new ConcurrentLinkedQueue<>();
private final Semaphore pendingLookupRequestSemaphore;
private final Semaphore maxLookupRequestSemaphore;
private final EventLoopGroup eventLoopGroup;
@@ -154,17 +154,37 @@ public class ClientCnx extends PulsarHandler {
None, SentConnectFrame, Ready, Failed, Connecting
}
-static class RequestTime {
-long creationTimeMs;
-long requestId;
+private static class RequestTime {
+final long creationTimeMs;
+final long requestId;
+final RequestType requestType;
+final ConcurrentLongHashMap> pendingRequestsMap;
-public RequestTime(long creationTime, long requestId) {
-super();
+RequestTime(long creationTime, long requestId, RequestType
requestType, ConcurrentLongHashMap> pendingRequestsMap) {
this.creationTimeMs = creationTime;
this.requestId = requestId;
+this.requestType = requestType;
+this.pendingRequestsMap = pendingRequestsMap;
}
}
+private enum RequestType {
+Command,
+GetLastMessageId,
+GetTopics,
+GetSchema,
+GetOrCreateSchema;
+
+String getDescription() {
+if (this == Command) {
+return "request";
+} else {
+return name() + " request";
+}
+}
+}
+
+
public ClientCnx(ClientConfigurationData conf, EventLoopGroup
eventLoopGroup) {
this(conf, eventLoopGroup, Commands.getCurrentProtocolVersion());
}
@@ -240,6 +260,7 @@ public class ClientCnx extends PulsarHandler {
pendingGetLastMessageIdRequests.forEach((key, future) ->
future.completeExceptionally(e));
pendingGetTopicsRequests.forEach((key, future) ->
future.completeExceptionally(e));
pendingGetSchemaRequests.forEach((key, future) ->
future.completeExceptionally(e));
+pendingGetOrCreateSchemaRequests.forEach((key, future) ->
future.completeExceptionally(e));
// Notify all attached producers/consumers so they have a chance to
reconnect
producers.forEach
[pulsar] branch master updated: Change message count in send callback to local variable (#8141)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new bb89e79 Change message count in send callback to local variable
(#8141)
bb89e79 is described below
commit bb89e79cf9dc8a07233b30bdf8de319c9a5d3b60
Author: Yunze Xu
AuthorDate: Sun Sep 27 17:09:28 2020 +0800
Change message count in send callback to local variable (#8141)
Fixes #8128
### Motivation
The send callback of `BatchMessageTest` is modifing a global variable which
is the count of successfully produced messages. So some tests may failed in
multi-thread environment.
### Modifications
- Add a `MessageCountSendCallback` class to replace the current send
callback function
---
pulsar-client-cpp/tests/BatchMessageTest.cc | 64 -
1 file changed, 35 insertions(+), 29 deletions(-)
diff --git a/pulsar-client-cpp/tests/BatchMessageTest.cc
b/pulsar-client-cpp/tests/BatchMessageTest.cc
index 592136e..4b7f23f 100644
--- a/pulsar-client-cpp/tests/BatchMessageTest.cc
+++ b/pulsar-client-cpp/tests/BatchMessageTest.cc
@@ -16,6 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
+#include
#include
#include
#include
@@ -42,7 +43,6 @@ DECLARE_LOG_OBJECT();
using namespace pulsar;
-static int globalTestBatchMessagesCounter = 0;
static int globalCount = 0;
static std::string lookupUrl = "pulsar://localhost:6650";
static std::string adminUrl = "http://localhost:8080/;;
@@ -55,10 +55,19 @@ static void messageListenerFunction(Consumer consumer,
const Message& msg) {
consumer.acknowledge(msg);
}
-static void sendCallBack(Result r, const MessageId& msgId) {
-ASSERT_EQ(r, ResultOk);
-globalTestBatchMessagesCounter++;
-}
+class MessageCountSendCallback {
+ public:
+MessageCountSendCallback(std::atomic_int& numOfMessagesProduced)
+: numOfMessagesProduced_(numOfMessagesProduced) {}
+
+void operator()(Result result, const MessageId&) {
+ASSERT_EQ(result, ResultOk);
+numOfMessagesProduced_++;
+}
+
+ private:
+std::atomic_int& numOfMessagesProduced_;
+};
static void sendFailCallBack(Result r, Result expect_result) { EXPECT_EQ(r,
expect_result); }
@@ -175,7 +184,6 @@ TEST(BatchMessageTest, testProducerTimeout) {
TEST(BatchMessageTest, testBatchSizeInBytes) {
std::string testName = std::to_string(epochTime) + "testBatchSizeInBytes";
-globalTestBatchMessagesCounter = 0;
Client client(lookupUrl);
std::string topicName = "persistent://public/default/" + testName;
@@ -216,12 +224,13 @@ TEST(BatchMessageTest, testBatchSizeInBytes) {
ProducerStatsImplPtr producerStatsImplPtr =
PulsarFriend::getProducerStatsPtr(producer);
// Send Asynchronously
+std::atomic_int numOfMessagesProduced{0};
std::string prefix = "12345678";
for (int i = 0; i < numOfMessages; i++) {
std::string messageContent = prefix + std::to_string(i);
Message msg =
MessageBuilder().setContent(messageContent).setProperty("msgIndex",
std::to_string(i)).build();
-producer.sendAsync(msg, );
+producer.sendAsync(msg,
MessageCountSendCallback(numOfMessagesProduced));
ASSERT_EQ(producerStatsImplPtr->getNumMsgsSent(), i + 1);
ASSERT_LT(PulsarFriend::sum(producerStatsImplPtr->getSendMap()), i +
1);
ASSERT_EQ(producerStatsImplPtr->getTotalMsgsSent(), i + 1);
@@ -246,7 +255,7 @@ TEST(BatchMessageTest, testBatchSizeInBytes) {
ASSERT_EQ(PulsarFriend::sum(producerStatsImplPtr->getTotalSendMap()),
numOfMessages);
// Number of messages produced
-ASSERT_EQ(globalTestBatchMessagesCounter, numOfMessages);
+ASSERT_EQ(numOfMessagesProduced.load(), numOfMessages);
// Number of messages consumed
ASSERT_EQ(i, numOfMessages);
@@ -254,7 +263,6 @@ TEST(BatchMessageTest, testBatchSizeInBytes) {
TEST(BatchMessageTest, testSmallReceiverQueueSize) {
std::string testName = std::to_string(epochTime) +
"testSmallReceiverQueueSize";
-globalTestBatchMessagesCounter = 0;
ClientConfiguration clientConf;
clientConf.setStatsIntervalInSeconds(20);
@@ -303,12 +311,13 @@ TEST(BatchMessageTest, testSmallReceiverQueueSize) {
ProducerStatsImplPtr producerStatsImplPtr =
PulsarFriend::getProducerStatsPtr(producer);
// Send Asynchronously
+std::atomic_int numOfMessagesProduced{0};
std::string prefix = testName;
for (int i = 0; i < numOfMessages; i++) {
std::string messageContent = prefix + std::to_string(i);
Message msg =
MessageBuilder().setContent(messageContent).setProperty("msgIndex",
std::to_s
[pulsar] branch master updated (f2933f7 -> fecd871)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from f2933f7 Always use SNI for TLS enabled Pulsar Java client. (#8117) add fecd871 Replace com.ning.asynchttpclient with org.asynchttpclient (#8099) No new revisions were added by this update. Summary of changes: pulsar-sql/presto-distribution/LICENSE | 3 +-- pulsar-sql/presto-distribution/pom.xml | 6 +++--- 2 files changed, 4 insertions(+), 5 deletions(-)
[pulsar] branch branch-2.5 updated: [Issue 8093]Fix client lookup hangs when broker restarts (#8101)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch branch-2.5
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.5 by this push:
new 61a7bd6 [Issue 8093]Fix client lookup hangs when broker restarts
(#8101)
61a7bd6 is described below
commit 61a7bd6e86078041037bd2cca87e3f28d82f4efa
Author: 4onni
AuthorDate: Thu Sep 24 15:03:59 2020 +0800
[Issue 8093]Fix client lookup hangs when broker restarts (#8101)
Fixes #8093
Client hangs forever when all brokers stop and then restart.
There are several steps need to be finished before the broker can be fully
started, as illustrated in the pseudo code below:
```
PulsarService#start():
broker.start(); // Step 1
webService.start(); // Step 2
leaderElectionService.start(); //Step 3
```
If a lookup request gets in between Step 2 and Step 3, a NPE would be
thrown, which will block all other coming requests from getting processed
properly.
Client can only connect to the broker after the election service started
successfully
This change added tests and can be verified as follows:
- * Added 2 test cases under `LeaderElectionServiceTest`
(cherry picked from commit 65cf9c095ea20c0a546e77458274848094131163)
---
.../broker/loadbalance/LeaderElectionService.java | 8 +
.../pulsar/broker/namespace/NamespaceService.java | 5 +
.../loadbalance/LeaderElectionServiceTest.java | 192 +
3 files changed, 205 insertions(+)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/LeaderElectionService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/LeaderElectionService.java
index eb9747f..8904b4f 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/LeaderElectionService.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/LeaderElectionService.java
@@ -54,6 +54,8 @@ public class LeaderElectionService {
private boolean stopped = true;
+private boolean elected = false;
+
private final ZooKeeper zkClient;
private final AtomicReference currentLeader = new
AtomicReference();
@@ -118,6 +120,7 @@ public class LeaderElectionService {
LeaderBroker leaderBroker = jsonMapper.readValue(data,
LeaderBroker.class);
currentLeader.set(leaderBroker);
isLeader.set(false);
+elected = true;
leaderListener.brokerIsAFollowerNow();
// If broker comes here it is a follower. Do nothing, wait for the
watch to trigger
@@ -135,6 +138,7 @@ public class LeaderElectionService {
// Update the current leader and set the flag to true
currentLeader.set(new
LeaderBroker(leaderBroker.getServiceUrl()));
isLeader.set(true);
+elected = true;
// Notify the listener that this broker is now the leader so
that it can collect usage and start load
// manager.
@@ -200,4 +204,8 @@ public class LeaderElectionService {
return isLeader.get();
}
+public boolean isElected() {
+return elected;
+}
+
}
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
index 039ea6b..c7bf68f 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
@@ -385,6 +385,11 @@ public class NamespaceService {
private void searchForCandidateBroker(NamespaceBundle bundle,
CompletableFuture> lookupFuture, boolean
authoritative) {
+if( null == pulsar.getLeaderElectionService() || !
pulsar.getLeaderElectionService().isElected()) {
+LOG.warn("The leader election has not yet been completed!
NamespaceBundle[{}]", bundle);
+lookupFuture.completeExceptionally(new IllegalStateException("The
leader election has not yet been completed!"));
+return;
+}
String candidateBroker = null;
try {
// check if this is Heartbeat or SLAMonitor namespace
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/LeaderElectionServiceTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/LeaderElectionServiceTest.java
new file mode 100644
index 000..4691b0e
--- /dev/null
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/LeaderElectionServiceTest.java
@@ -0,0 +1,192 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for
[pulsar] branch branch-2.6 updated: [Issue 8093]Fix client lookup hangs when broker restarts (#8101)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch branch-2.6
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.6 by this push:
new a4a363c [Issue 8093]Fix client lookup hangs when broker restarts
(#8101)
a4a363c is described below
commit a4a363cb3eeff7f76466fe078efe6fb7c793d136
Author: 4onni
AuthorDate: Thu Sep 24 15:03:59 2020 +0800
[Issue 8093]Fix client lookup hangs when broker restarts (#8101)
Fixes #8093
### Motivation
Client hangs forever when all brokers stop and then restart.
There are several steps need to be finished before the broker can be fully
started, as illustrated in the pseudo code below:
```
PulsarService#start():
broker.start(); // Step 1
webService.start(); // Step 2
leaderElectionService.start(); //Step 3
```
If a lookup request gets in between Step 2 and Step 3, a NPE would be
thrown, which will block all other coming requests from getting processed
properly.
### Modifications
Client can only connect to the broker after the election service started
successfully
### Verifying this change
This change added tests and can be verified as follows:
- * Added 2 test cases under `LeaderElectionServiceTest`
(cherry picked from commit 65cf9c095ea20c0a546e77458274848094131163)
---
.../broker/loadbalance/LeaderElectionService.java | 8 +
.../pulsar/broker/namespace/NamespaceService.java | 5 +
.../loadbalance/LeaderElectionServiceTest.java | 192 +
3 files changed, 205 insertions(+)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/LeaderElectionService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/LeaderElectionService.java
index 59c650e..ad54feb 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/LeaderElectionService.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/LeaderElectionService.java
@@ -54,6 +54,8 @@ public class LeaderElectionService {
private boolean stopped = true;
+private boolean elected = false;
+
private final ZooKeeper zkClient;
private final AtomicReference currentLeader = new
AtomicReference();
@@ -118,6 +120,7 @@ public class LeaderElectionService {
LeaderBroker leaderBroker = jsonMapper.readValue(data,
LeaderBroker.class);
currentLeader.set(leaderBroker);
isLeader.set(false);
+elected = true;
leaderListener.brokerIsAFollowerNow();
// If broker comes here it is a follower. Do nothing, wait for the
watch to trigger
@@ -135,6 +138,7 @@ public class LeaderElectionService {
// Update the current leader and set the flag to true
currentLeader.set(new
LeaderBroker(leaderBroker.getServiceUrl()));
isLeader.set(true);
+elected = true;
// Notify the listener that this broker is now the leader so
that it can collect usage and start load
// manager.
@@ -200,4 +204,8 @@ public class LeaderElectionService {
return isLeader.get();
}
+public boolean isElected() {
+return elected;
+}
+
}
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
index 6c8d971..62b48db 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java
@@ -400,6 +400,11 @@ public class NamespaceService {
private void searchForCandidateBroker(NamespaceBundle bundle,
CompletableFuture> lookupFuture,
LookupOptions options) {
+if( null == pulsar.getLeaderElectionService() || !
pulsar.getLeaderElectionService().isElected()) {
+LOG.warn("The leader election has not yet been completed!
NamespaceBundle[{}]", bundle);
+lookupFuture.completeExceptionally(new IllegalStateException("The
leader election has not yet been completed!"));
+return;
+}
String candidateBroker = null;
boolean authoritativeRedirect =
pulsar.getLeaderElectionService().isLeader();
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/LeaderElectionServiceTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/LeaderElectionServiceTest.java
new file mode 100644
index 000..4691b0e
--- /dev/null
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/LeaderElectionServiceTest.java
@@ -0,0 +1,192 @@
+/**
+ *
svn commit: r41545 - /dev/pulsar/pulsar-manager/pulsar-manager-0.2.0/ /release/pulsar/pulsar-manager/pulsar-manager-0.2.0/
Author: zhaijia Date: Wed Sep 23 01:49:24 2020 New Revision: 41545 Log: pulsar manager 0.2.0 Added: release/pulsar/pulsar-manager/pulsar-manager-0.2.0/ - copied from r41544, dev/pulsar/pulsar-manager/pulsar-manager-0.2.0/ Removed: dev/pulsar/pulsar-manager/pulsar-manager-0.2.0/
[pulsar] branch master updated: Added support to force deleting namespace (#7993)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 7505dac Added support to force deleting namespace (#7993)
7505dac is described below
commit 7505dacf2f731d69d42ca02eb3ce6a2159816995
Author: Fangbin Sun
AuthorDate: Tue Sep 22 07:12:38 2020 +0800
Added support to force deleting namespace (#7993)
### Motivation
Fixes #7959
### Modifications
Add a optional field to force the deletion of all stuffs related to
namespace.
---
.../pulsar/broker/admin/impl/NamespacesBase.java | 234 +
.../apache/pulsar/broker/admin/v1/Namespaces.java | 6 +-
.../apache/pulsar/broker/admin/v2/Namespaces.java | 6 +-
.../org/apache/pulsar/broker/admin/AdminTest.java | 2 +-
.../apache/pulsar/broker/admin/NamespacesTest.java | 24 +--
.../org/apache/pulsar/client/admin/Namespaces.java | 72 +++
.../client/admin/internal/NamespacesImpl.java | 40
.../pulsar/admin/cli/PulsarAdminToolTest.java | 2 +-
.../org/apache/pulsar/admin/cli/CmdNamespaces.java | 8 +-
9 files changed, 374 insertions(+), 20 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
index a1d7e3f..030ebe1 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
@@ -97,6 +97,7 @@ import org.apache.pulsar.common.util.FutureUtil;
import org.apache.pulsar.common.util.ObjectMapperFactory;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.KeeperException.NoNodeException;
+import org.apache.zookeeper.ZKUtil;
import org.apache.zookeeper.data.Stat;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -145,6 +146,14 @@ public abstract class NamespacesBase extends AdminResource
{
}
}
+protected void internalDeleteNamespace(AsyncResponse asyncResponse,
boolean authoritative, boolean force) {
+if (force) {
+internalDeleteNamespaceForcefully(asyncResponse, authoritative);
+} else {
+internalDeleteNamespace(asyncResponse, authoritative);
+}
+}
+
@SuppressWarnings("deprecation")
protected void internalDeleteNamespace(AsyncResponse asyncResponse,
boolean authoritative) {
validateTenantOperation(namespaceName.getTenant(),
TenantOperation.DELETE_NAMESPACE);
@@ -310,6 +319,167 @@ public abstract class NamespacesBase extends
AdminResource {
}
@SuppressWarnings("deprecation")
+protected void internalDeleteNamespaceForcefully(AsyncResponse
asyncResponse, boolean authoritative) {
+validateTenantOperation(namespaceName.getTenant(),
TenantOperation.DELETE_NAMESPACE);
+validatePoliciesReadOnlyAccess();
+
+// ensure that non-global namespace is directed to the correct cluster
+if (!namespaceName.isGlobal()) {
+validateClusterOwnership(namespaceName.getCluster());
+}
+
+Entry policiesNode = null;
+Policies policies = null;
+
+// ensure the local cluster is the only cluster for the global
namespace configuration
+try {
+policiesNode = policiesCache().getWithStat(path(POLICIES,
namespaceName.toString())).orElseThrow(
+() -> new RestException(Status.NOT_FOUND, "Namespace " +
namespaceName + " does not exist."));
+
+policies = policiesNode.getKey();
+if (namespaceName.isGlobal()) {
+if (policies.replication_clusters.size() > 1) {
+// There are still more than one clusters configured for
the global namespace
+throw new RestException(Status.PRECONDITION_FAILED,
"Cannot delete the global namespace "
++ namespaceName + ". There are still more than one
replication clusters configured.");
+}
+if (policies.replication_clusters.size() == 1
+&&
!policies.replication_clusters.contains(config().getClusterName())) {
+// the only replication cluster is other cluster, redirect
+String replCluster =
Lists.newArrayList(policies.replication_clusters).get(0);
+ClusterData replClusterData =
clustersCache().get(AdminResource.path("clusters", replCluster))
+.orElseThrow(() -> new
RestException(Status.NOT_FOUND,
+"Cluster " + replCluster + " does not
exist"));
+
[pulsar] branch master updated (d3c0ccb -> f75289d)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from d3c0ccb [pulsar-client] support input-stream for trustStore cert (#7442) add f75289d Add Snappy library to Docker images for building C++ packages (#8086) No new revisions were added by this update. Summary of changes: pulsar-client-cpp/pkg/deb/Dockerfile | 8 pulsar-client-cpp/pkg/rpm/Dockerfile | 8 2 files changed, 16 insertions(+)
[pulsar] branch master updated (e78a864 -> 41c4f44)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from e78a864 SinkRecord adds an overridden method (#8038) add 41c4f44 Change bookkeeper main method class (#8065) No new revisions were added by this update. Summary of changes: bin/bookkeeper | 2 +- bin/pulsar | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
[pulsar] branch master updated (609d3d4 -> e78a864)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 609d3d4 transaction flaky test (#8071) add e78a864 SinkRecord adds an overridden method (#8038) No new revisions were added by this update. Summary of changes: .../java/org/apache/pulsar/functions/instance/SinkRecord.java | 10 ++ 1 file changed, 10 insertions(+)
[pulsar] branch master updated: Cancel retry if the error is autheticate failed (#8058)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new bf96619 Cancel retry if the error is autheticate failed (#8058)
bf96619 is described below
commit bf96619ae96655a27baff15551305fa7cbcb2835
Author: Yong Zhang
AuthorDate: Wed Sep 16 21:39:56 2020 +0800
Cancel retry if the error is autheticate failed (#8058)
---
Fixes #7929
*Motivation*
We shouldn't retry to connect to the server if the client has authenticate
error.
---
.../main/java/org/apache/pulsar/client/impl/ClientCnx.java | 4
.../org/apache/pulsar/client/impl/PulsarClientImpl.java| 4 +++-
.../auth/token/PulsarTokenAuthenticationBaseSuite.java | 14 ++
3 files changed, 21 insertions(+), 1 deletion(-)
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
index b32dc99..6f8c511 100644
--- a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
+++ b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java
@@ -607,6 +607,10 @@ public class ClientCnx extends PulsarHandler {
log.warn("{} Producer creation has been blocked because backlog
quota exceeded for producer topic",
ctx.channel());
}
+if (error.getError() == ServerError.AuthenticationError) {
+connectionFuture.completeExceptionally(new
PulsarClientException.AuthenticationException(error.getMessage()));
+log.error("{} Failed to authenticate the client", ctx.channel());
+}
CompletableFuture requestFuture =
pendingRequests.remove(requestId);
if (requestFuture != null) {
requestFuture.completeExceptionally(getPulsarClientException(error.getError(),
error.getMessage()));
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PulsarClientImpl.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PulsarClientImpl.java
index 6c4dced..c33eb9b 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PulsarClientImpl.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PulsarClientImpl.java
@@ -704,7 +704,9 @@ public class PulsarClientImpl implements PulsarClient {
lookup.getPartitionedTopicMetadata(topicName).thenAccept(future::complete).exceptionally(e
-> {
long nextDelay = Math.min(backoff.next(), remainingTime.get());
// skip retry scheduler when set lookup throttle in client or
server side which will lead to `TooManyRequestsException`
-boolean isLookupThrottling =
!PulsarClientException.isRetriableError(e.getCause()) || e.getCause()
instanceof PulsarClientException.TooManyRequestsException;
+boolean isLookupThrottling =
!PulsarClientException.isRetriableError(e.getCause())
+|| e.getCause() instanceof
PulsarClientException.TooManyRequestsException
+|| e.getCause() instanceof
PulsarClientException.AuthenticationException;
if (nextDelay <= 0 || isLookupThrottling) {
future.completeExceptionally(e);
return null;
diff --git
a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/auth/token/PulsarTokenAuthenticationBaseSuite.java
b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/auth/token/PulsarTokenAuthenticationBaseSuite.java
index ad0fa83..7a923ee 100644
---
a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/auth/token/PulsarTokenAuthenticationBaseSuite.java
+++
b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/auth/token/PulsarTokenAuthenticationBaseSuite.java
@@ -329,4 +329,18 @@ public abstract class PulsarTokenAuthenticationBaseSuite
extends PulsarClusterTe
}
}
}
+
+@Test
+public void testAuthenticationFailedImmediately() throws
PulsarClientException {
+try {
+@Cleanup
+PulsarClient client = PulsarClient.builder()
+.serviceUrl(pulsarCluster.getPlainTextServiceUrl())
+.authentication(AuthenticationFactory.token("invalid_token"))
+.build();
+client.newProducer().topic("test_token_topic" + randomName(4));
+} catch (PulsarClientException.AuthenticationException pae) {
+// expected error
+}
+}
}
[pulsar] branch master updated (77fca3e -> 406ad72)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 77fca3e [Transaction] Fix transaction message ack (#8007) add 406ad72 Update the jwt authentication doc (#8073) No new revisions were added by this update. Summary of changes: site2/docs/security-jwt.md | 8 site2/website/versioned_docs/version-2.6.0/security-jwt.md | 8 site2/website/versioned_docs/version-2.6.1/security-jwt.md | 8 site2/website/versioned_docs/version-2.6.2/security-jwt.md | 8 4 files changed, 32 insertions(+)
[pulsar] branch master updated (77fca3e -> 406ad72)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 77fca3e [Transaction] Fix transaction message ack (#8007) add 406ad72 Update the jwt authentication doc (#8073) No new revisions were added by this update. Summary of changes: site2/docs/security-jwt.md | 8 site2/website/versioned_docs/version-2.6.0/security-jwt.md | 8 site2/website/versioned_docs/version-2.6.1/security-jwt.md | 8 site2/website/versioned_docs/version-2.6.2/security-jwt.md | 8 4 files changed, 32 insertions(+)
[pulsar] branch master updated (77fca3e -> 406ad72)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 77fca3e [Transaction] Fix transaction message ack (#8007) add 406ad72 Update the jwt authentication doc (#8073) No new revisions were added by this update. Summary of changes: site2/docs/security-jwt.md | 8 site2/website/versioned_docs/version-2.6.0/security-jwt.md | 8 site2/website/versioned_docs/version-2.6.1/security-jwt.md | 8 site2/website/versioned_docs/version-2.6.2/security-jwt.md | 8 4 files changed, 32 insertions(+)
[pulsar] branch master updated (f06f0d8 -> fd2319f)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from f06f0d8 Fix the failed ci issue (#8032) add fd2319f [Doc]--Update Pulsar Admin CLI doc to support to set/get/remove persistence polices, message TTL, and backlog quota on topic level (#7852) No new revisions were added by this update. Summary of changes: site2/docs/concepts-multi-tenancy.md | 15 ++ site2/docs/reference-pulsar-admin.md | 95 2 files changed, 110 insertions(+)
[pulsar] branch master updated: Fix the failed ci issue (#8032)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new f06f0d8 Fix the failed ci issue (#8032) f06f0d8 is described below commit f06f0d8baf3975367c3f5a14dc5a0bda692487ba Author: Yong Zhang AuthorDate: Mon Sep 14 11:12:29 2020 +0800 Fix the failed ci issue (#8032) **Motivation** Currently, the Pulsar CI always failed by: ``` 2020-09-11T00:49:20.3402047Z [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.1.0:add-source (default) on project flink: Execution default of goal net.alchim31.maven:scala-maven-plugin:4.1.0:add-source failed: Plugin net.alchim31.maven:scala-maven-plugin:4.1.0 or one of its dependencies could not be resolved: The following artifacts could not be resolved: org.apache.maven:maven-compat:jar:3.3.9, org.apache.maven:maven-builder-support:jar:3.3.9, org.apache.maven:m [...] ``` Upgrade the scala-maven-plugin to resolve this issue. --- examples/flink/pom.xml | 2 +- examples/spark/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/examples/flink/pom.xml b/examples/flink/pom.xml index 29869b6..23f5b00 100644 --- a/examples/flink/pom.xml +++ b/examples/flink/pom.xml @@ -131,7 +131,7 @@ net.alchim31.maven scala-maven-plugin -4.1.0 +4.3.0 diff --git a/examples/spark/pom.xml b/examples/spark/pom.xml index 2aef2d1..81abf28 100644 --- a/examples/spark/pom.xml +++ b/examples/spark/pom.xml @@ -98,7 +98,7 @@ net.alchim31.maven scala-maven-plugin -4.1.0 +4.3.0
[pulsar] branch master updated (4cbbea1 -> ff6876c)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 4cbbea1 Fix no return value of GetMaxProducers in topic cli (#7990) add ff6876c [cpp-client] Fix for not respecting custom LoggerFactory client config (#7932) No new revisions were added by this update. Summary of changes: pulsar-client-cpp/lib/ClientImpl.cc | 18 pulsar-client-cpp/lib/LogUtils.cc | 2 + pulsar-client-cpp/lib/LogUtils.h| 2 + pulsar-client-cpp/tests/CustomLoggerTest.cc | 69 + 4 files changed, 83 insertions(+), 8 deletions(-) create mode 100644 pulsar-client-cpp/tests/CustomLoggerTest.cc
[pulsar] branch master updated (fe621a6 -> 22b8923)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from fe621a6 make bookkeeper throttle Configurable (#7901) add 22b8923 Add config to lazily recover cursors when recovering a managed ledger. (#7858) No new revisions were added by this update. Summary of changes: conf/broker.conf | 6 ++ deployment/terraform-ansible/templates/broker.conf | 6 ++ .../bookkeeper/mledger/ManagedLedgerConfig.java| 20 + .../bookkeeper/mledger/impl/ManagedLedgerImpl.java | 86 +++--- .../bookkeeper/mledger/impl/ManagedLedgerTest.java | 35 + .../apache/pulsar/broker/ServiceConfiguration.java | 6 ++ .../pulsar/broker/service/BrokerService.java | 1 + site2/docs/reference-configuration.md | 2 +- 8 files changed, 136 insertions(+), 26 deletions(-)
[pulsar] branch master updated (e72b0f7 -> fe621a6)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from e72b0f7 Removed blocking call (#7983) add fe621a6 make bookkeeper throttle Configurable (#7901) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/pulsar/broker/ServiceConfiguration.java| 6 ++ .../java/org/apache/pulsar/broker/BookKeeperClientFactoryImpl.java | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-)
[pulsar] branch master updated: Removed blocking call (#7983)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new e72b0f7 Removed blocking call (#7983)
e72b0f7 is described below
commit e72b0f78e15cad54378564be0c772fda36009364
Author: David Kjerrumgaard <35466513+david-stream...@users.noreply.github.com>
AuthorDate: Wed Sep 9 05:14:41 2020 -0700
Removed blocking call (#7983)
Co-authored-by: David Kjerrumgaard
Motivation
Explain here the context, and why you're making that change. What is the
problem you're trying to solve.
Modifications
Describe the modifications you've done.
---
.../apache/pulsar/functions/instance/JavaInstanceRunnable.java| 8 +---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
index 45a8174..f2cfbf3 100644
---
a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
+++
b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
@@ -34,12 +34,14 @@ import net.jodah.typetools.TypeResolver;
import org.apache.bookkeeper.api.StorageClient;
import org.apache.bookkeeper.api.kv.Table;
import org.apache.bookkeeper.clients.StorageClientBuilder;
+import org.apache.bookkeeper.clients.admin.SimpleStorageAdminClientImpl;
import org.apache.bookkeeper.clients.admin.StorageAdminClient;
import org.apache.bookkeeper.clients.config.StorageClientSettings;
import org.apache.bookkeeper.clients.exceptions.ClientException;
import org.apache.bookkeeper.clients.exceptions.InternalServerException;
import org.apache.bookkeeper.clients.exceptions.NamespaceNotFoundException;
import org.apache.bookkeeper.clients.exceptions.StreamNotFoundException;
+import org.apache.bookkeeper.clients.utils.ClientResources;
import org.apache.bookkeeper.common.util.Backoff.Jitter;
import org.apache.bookkeeper.common.util.Backoff.Jitter.Type;
import org.apache.bookkeeper.stream.proto.NamespaceConfiguration;
@@ -331,9 +333,9 @@ public class JavaInstanceRunnable implements AutoCloseable,
Runnable {
}
private void createStateTable(String tableNs, String tableName,
StorageClientSettings settings) throws Exception {
-try (StorageAdminClient storageAdminClient =
StorageClientBuilder.newBuilder()
-.withSettings(settings)
-.buildAdmin()) {
+ try (StorageAdminClient storageAdminClient = new
SimpleStorageAdminClientImpl(
+
StorageClientSettings.newBuilder().serviceUri(stateStorageServiceUrl).build(),
+ ClientResources.create().scheduler())){
StreamConfiguration streamConf =
StreamConfiguration.newBuilder(DEFAULT_STREAM_CONF)
.setInitialNumRanges(4)
.setMinNumRanges(4)
[pulsar] branch master updated: pulsar-perf support setting message key (#7989)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 2773e61 pulsar-perf support setting message key (#7989)
2773e61 is described below
commit 2773e6188e1ab5b4549b900328f12c94fd514b3a
Author: feynmanlin
AuthorDate: Wed Sep 9 20:12:03 2020 +0800
pulsar-perf support setting message key (#7989)
Fixes #7896
### Motivation
In the key-shared mode, only one consumer can consume the message.
Therefore, pulsar-perf should support setting message key
### Modifications
Added a parameter, user can set the message key generation mode: `radom` or
`autoIncrement`
### Verifying this change
org.apache.pulsar.testclient.PerformanceProducerTest
* support setting routing key
* add unit test
* set msg num
* close consumer
* close consumer
---
pulsar-testclient/pom.xml | 8 ++
.../pulsar/testclient/PerformanceProducer.java | 26 +++-
.../pulsar/testclient/PerformanceProducerTest.java | 146 +
3 files changed, 179 insertions(+), 1 deletion(-)
diff --git a/pulsar-testclient/pom.xml b/pulsar-testclient/pom.xml
index 132f327..21c293b 100644
--- a/pulsar-testclient/pom.xml
+++ b/pulsar-testclient/pom.xml
@@ -72,6 +72,14 @@
+ ${project.groupId}
+ pulsar-broker
+ ${project.version}
+ test-jar
+ test
+
+
+
commons-configuration
commons-configuration
diff --git
a/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceProducer.java
b/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceProducer.java
index 2f64c18..812cf0d 100644
---
a/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceProducer.java
+++
b/pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceProducer.java
@@ -207,6 +207,10 @@ public class PerformanceProducer {
@Parameter(names = { "-ef",
"--exit-on-failure" }, description = "Exit from the process on
publish failure (default: disable)")
public boolean exitOnFailure = false;
+
+@Parameter(names = {"-mk", "--message-key-generation-mode"},
description = "The generation mode of message key" +
+", valid options are: [autoIncrement, random]")
+public String messageKeyGenerationMode = null;
}
static class EncKeyReader implements CryptoKeyReader {
@@ -288,6 +292,9 @@ public class PerformanceProducer {
if (isBlank(arguments.tlsTrustCertsFilePath)) {
arguments.tlsTrustCertsFilePath =
prop.getProperty("tlsTrustCertsFilePath", "");
}
+if (isBlank(arguments.messageKeyGenerationMode)) {
+arguments.messageKeyGenerationMode =
prop.getProperty("messageKeyGenerationMode", null);
+}
if (arguments.tlsAllowInsecureConnection == null) {
arguments.tlsAllowInsecureConnection =
Boolean.parseBoolean(prop
.getProperty("tlsAllowInsecureConnection", ""));
@@ -494,7 +501,14 @@ public class PerformanceProducer {
long startTime = System.nanoTime();
long warmupEndTime = startTime + (long)
(arguments.warmupTimeSeconds * 1e9);
long testEndTime = startTime + (long) (arguments.testTime * 1e9);
-
+MessageKeyGenerationMode msgKeyMode = null;
+if (isNotBlank(arguments.messageKeyGenerationMode)) {
+try {
+msgKeyMode =
MessageKeyGenerationMode.valueOf(arguments.messageKeyGenerationMode);
+} catch (IllegalArgumentException e) {
+throw new
IllegalArgumentException("messageKeyGenerationMode only support [autoIncrement,
random]");
+}
+}
// Send messages on all topics/producers
long totalSent = 0;
while (true) {
@@ -535,6 +549,12 @@ public class PerformanceProducer {
if (arguments.delay >0) {
messageBuilder.deliverAfter(arguments.delay,
TimeUnit.SECONDS);
}
+//generate msg key
+if (msgKeyMode == MessageKeyGenerationMode.random) {
+messageBuilder.key(String.valueOf(random.nextInt()));
+} else if (msgKeyMode ==
MessageKeyGenerationMode.autoIncrement) {
+
[pulsar] branch master updated: Add Neutron & Supernova (#7992)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new e84ec79 Add Neutron & Supernova (#7992) e84ec79 is described below commit e84ec79ef28fca1c2f23c12a1df1f5ac3e743f7e Author: Gabriel Volpe AuthorDate: Wed Sep 9 13:49:09 2020 +0200 Add Neutron & Supernova (#7992) Add new clients we maintain at Chatroulette. --- site2/docs/getting-started-clients.md | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/site2/docs/getting-started-clients.md b/site2/docs/getting-started-clients.md index 6899143..d1c218f 100644 --- a/site2/docs/getting-started-clients.md +++ b/site2/docs/getting-started-clients.md @@ -26,7 +26,9 @@ Besides the official released clients, multiple projects on developing Pulsar cl | Language | Project | Maintainer | License | Description | |--|-||-|-| | Go | [pulsar-client-go](https://github.com/Comcast/pulsar-client-go) | [Comcast](https://github.com/Comcast) | [](https://opensource.org/licenses/Apache-2.0) | A native golang client | -| Go | [go-pulsar](https://github.com/t2y/go-pulsar) | [t2y](https://github.com/t2y) | [](https://opensource.org/licenses/Apache-2.0) | | +| Go | [go-pulsar](https://github.com/t2y/go-pulsar) | [t2y](https://github.com/t2y) | [](https://opensource.org/licenses/Apache-2.0) | +| Haskell | [supernova](https://github.com/cr-org/supernova) | [Chatroulette](https://github.com/cr-org) | [](https://opensource.org/licenses/Apache-2.0) | Native Pulsar client for Haskell | +| Scala | [neutron](https://github.com/cr-org/neutron) | [Chatroulette](https://github.com/cr-org) | [](https://opensource.org/licenses/Apache-2.0) | Purely functional Apache Pulsar client for Scala built on top of Fs2 | | Scala | [pulsar4s](https://github.com/sksamuel/pulsar4s) | [sksamuel](https://github.com/sksamuel) | [](https://opensource.org/licenses/Apache-2.0) | Idomatic, typesafe, and reactive Scala client for Apache Pulsar | | Rust | [pulsar-rs](https://github.com/wyyerd/pulsar-rs) | [Wyyerd Group](https://github.com/wyyerd) | [](https://opensource.org/licenses/Apache-2.0) | Future-based Rust bindings for Apache Pulsar | -| .NET | [pulsar-client-dotnet](https://github.com/fsharplang-ru/pulsar-client-dotnet) | [Lanayx](https://github.com/Lanayx) | [](https://opensource.org/licenses/MIT) | Native .NET client for C#/F#/VB | \ No newline at end of file +| .NET | [pulsar-client-dotnet](https://github.com/fsharplang-ru/pulsar-client-dotnet) | [Lanayx](https://github.com/Lanayx) | [](https://opensource.org/licenses/MIT) | Native .NET client for C#/F#/VB |
[pulsar] branch master updated (7154185 -> e7c4074)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git.
from 7154185 Fix comment: pulsar.Client.create_producer() (#7972)
add e7c4074 [C++] Support key based batching (#7996)
No new revisions were added by this update.
Summary of changes:
pulsar-client-cpp/include/pulsar/Message.h | 2 +-
.../include/pulsar/ProducerConfiguration.h | 32 +++-
pulsar-client-cpp/lib/BatchMessageContainer.cc | 201 +
pulsar-client-cpp/lib/BatchMessageContainer.h | 115 ++--
pulsar-client-cpp/lib/BatchMessageContainerBase.cc | 82 +
pulsar-client-cpp/lib/BatchMessageContainerBase.h | 167 +
.../lib/BatchMessageKeyBasedContainer.cc | 129 +
.../lib/BatchMessageKeyBasedContainer.h| 62 +++
pulsar-client-cpp/lib/ClientConnection.cc | 7 +-
pulsar-client-cpp/lib/ClientConnection.h | 5 +-
pulsar-client-cpp/lib/Commands.cc | 30 ++-
pulsar-client-cpp/lib/Commands.h | 4 +-
pulsar-client-cpp/lib/ConsumerImpl.cc | 2 +-
pulsar-client-cpp/lib/MessageAndCallbackBatch.cc | 66 +++
pulsar-client-cpp/lib/MessageAndCallbackBatch.h| 82 +
pulsar-client-cpp/lib/MessageCrypto.cc | 4 +-
pulsar-client-cpp/lib/MessageCrypto.h | 4 +-
.../pulsar/MessageBatch.h => lib/OpSendMsg.h} | 49 ++---
.../lib/{BoostHash.h => PendingFailures.h} | 31 ++--
pulsar-client-cpp/lib/ProducerConfiguration.cc | 17 +-
pulsar-client-cpp/lib/ProducerConfigurationImpl.h | 2 +
pulsar-client-cpp/lib/ProducerImpl.cc | 197
pulsar-client-cpp/lib/ProducerImpl.h | 32 +---
pulsar-client-cpp/tests/BatchMessageTest.cc| 3 +
pulsar-client-cpp/tests/KeyBasedBatchingTest.cc| 178 ++
pulsar-client-cpp/tests/KeySharedConsumerTest.cc | 45 -
26 files changed, 1130 insertions(+), 418 deletions(-)
create mode 100644 pulsar-client-cpp/lib/BatchMessageContainerBase.cc
create mode 100644 pulsar-client-cpp/lib/BatchMessageContainerBase.h
create mode 100644 pulsar-client-cpp/lib/BatchMessageKeyBasedContainer.cc
create mode 100644 pulsar-client-cpp/lib/BatchMessageKeyBasedContainer.h
create mode 100644 pulsar-client-cpp/lib/MessageAndCallbackBatch.cc
create mode 100644 pulsar-client-cpp/lib/MessageAndCallbackBatch.h
copy pulsar-client-cpp/{include/pulsar/MessageBatch.h => lib/OpSendMsg.h} (56%)
copy pulsar-client-cpp/lib/{BoostHash.h => PendingFailures.h} (66%)
create mode 100644 pulsar-client-cpp/tests/KeyBasedBatchingTest.cc
[pulsar] branch master updated (ed356ad -> 8933d8d)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from ed356ad Remove python2 from the pulsar images build (#7936) add 8933d8d Fix dispatchRate is overwritten (#8004) No new revisions were added by this update. Summary of changes: .../broker/service/persistent/PersistentTopic.java | 36 ++- .../pulsar/broker/admin/TopicPoliciesTest.java | 42 ++ .../broker/service/InactiveTopicDeleteTest.java| 6 ++-- 3 files changed, 65 insertions(+), 19 deletions(-)
[pulsar] branch master updated: Remove python2 from the pulsar images build (#7936)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new ed356ad Remove python2 from the pulsar images build (#7936)
ed356ad is described below
commit ed356add3bf5194058854b2fdccc739be6d697c6
Author: Yong Zhang
AuthorDate: Tue Sep 8 21:24:58 2020 +0800
Remove python2 from the pulsar images build (#7936)
The pulsar docker and pulsar standalone docker image build failed by:
```
Processing
./pulsar-client/pulsar_client-2.6.0-cp27-cp27mu-manylinux1_x86_64.whl
2020-08-25T02:23:28.9567844Z [INFO] Collecting fastavro
2020-08-25T02:23:29.0080316Z [INFO] Downloading fastavro-1.0.0.tar.gz
(658 kB)
2020-08-25T02:23:29.3108074Z [INFO] ERROR: Command errored out with
exit status 1:
2020-08-25T02:23:29.3109489Z [INFO] command: /usr/bin/python2.7 -c
'import sys, setuptools, tokenize; sys.argv[0] =
'"'"'/tmp/pip-install-jvZueR/fastavro/setup.py'"'"';
__file__='"'"'/tmp/pip-install-jvZueR/fastavro/setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
--egg-base /tmp/pip-pip-egg-info-iH1mMy
2020-08-25T02:23:29.3110061Z [INFO] cwd:
/tmp/pip-install-jvZueR/fastavro/
2020-08-25T02:23:29.3110303Z [INFO] Complete output (5 lines):
2020-08-25T02:23:29.3110520Z [INFO] Traceback (most recent call last):
2020-08-25T02:23:29.3110741Z [INFO] File "", line 1, in
2020-08-25T02:23:29.390Z [INFO] File
"/tmp/pip-install-jvZueR/fastavro/setup.py", line 58, in
2020-08-25T02:23:29.3111437Z [INFO] if sys.implementation.name !=
"pypy":
2020-08-25T02:23:29.3111886Z [INFO] AttributeError: 'module' object has
no attribute 'implementation'
```
I remove the python2.7 and using the python3 to build images.
* Remove python2 in images
* Remove python2 stuff
* Update the default python version
---
dashboard/Dockerfile| 2 +-
docker/pulsar-standalone/Dockerfile | 14 ++
docker/pulsar/Dockerfile| 6 ++
docker/pulsar/pom.xml | 16
4 files changed, 13 insertions(+), 25 deletions(-)
diff --git a/dashboard/Dockerfile b/dashboard/Dockerfile
index 22be50a..ab737d3 100644
--- a/dashboard/Dockerfile
+++ b/dashboard/Dockerfile
@@ -25,7 +25,7 @@ RUN bash -c "echo deb
http://apt.postgresql.org/pub/repos/apt/ stretch-pgdg main
RUN wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc |
apt-key add -
RUN apt-get update
-RUN apt-get -y install postgresql-11 postgresql-contrib libpq-dev python sudo
nginx supervisor
+RUN apt-get -y install postgresql-11 postgresql-contrib libpq-dev sudo nginx
supervisor
# Postgres configuration
COPY conf/postgresql.conf /etc/postgresql/11/main/
diff --git a/docker/pulsar-standalone/Dockerfile
b/docker/pulsar-standalone/Dockerfile
index 93a9876..8555af0 100644
--- a/docker/pulsar-standalone/Dockerfile
+++ b/docker/pulsar-standalone/Dockerfile
@@ -29,7 +29,10 @@ FROM openjdk:8-jdk
# Note that the libpq-dev package is needed here in order to install
# the required python psycopg2 package (for postgresql) later
RUN apt-get update \
-&& apt-get -y install python2.7 python2.7-dev python-pip postgresql sudo
nginx supervisor libpq-dev
+&& apt-get -y install python3.7 python3.7-dev python3-pip postgresql sudo
nginx supervisor libpq-dev
+
+RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10
+RUN ln -s /usr/bin/pip3 /usr/bin/pip
# Postgres configuration
COPY --from=dashboard /etc/postgresql/11/main/postgresql.conf
/etc/postgresql/11/main/postgresql.conf
@@ -59,15 +62,17 @@ COPY --from=dashboard /pulsar/django /pulsar/django
COPY --from=dashboard /pulsar/requirements.txt /pulsar/django
COPY --from=dashboard /pulsar/conf/* /pulsar/conf/
+RUN pip -V
+
# Python dependencies
-RUN pip2 install -r /pulsar/django/requirements.txt
+RUN pip install -r /pulsar/django/requirements.txt
# Copy web-app sources
# Setup database and create tables
COPY --from=dashboard /pulsar/init-postgres.sh /pulsar/django/init-postgres.sh
RUN mkdir /data
RUN /pulsar/django/init-postgres.sh
-RUN sudo -u postgres /etc/init.d/postgresql stop
+RUN sudo -u postgres /etc/init.d/postgresql stop
# Add postgresql to supervisord. Redirect logs to stdout
RUN echo "\n[program:postgresql]\n\
command = /etc/init.d/postgresql start\n\
@@ -78,9 +83,10 @@ redirect_stderr=true" >>
/etc/supervisor/conf.d/supervisor-app.conf
# Collect all static fil
[pulsar] branch master updated (8f2540f -> 5adbc1f)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 8f2540f [Transaction] Txn ack abort process on subscription (#7979) add 5adbc1f [Security] Upgrade the snakeyaml verion to 1.26 (#7994) No new revisions were added by this update. Summary of changes: distribution/server/src/assemble/LICENSE.bin.txt | 23 ++--- pom.xml| 4 +- pulsar-functions/runtime/pom.xml | 2 +- .../auth/KubernetesFunctionAuthProvider.java | 4 +- .../auth/KubernetesSecretsTokenAuthProvider.java | 34 ++-- .../BasicKubernetesManifestCustomizer.java | 2 +- .../kubernetes/KubernetesManifestCustomizer.java | 4 +- .../runtime/kubernetes/KubernetesRuntime.java | 60 +++--- .../kubernetes/KubernetesRuntimeFactory.java | 10 ++-- .../KubernetesSecretsTokenAuthProviderTest.java| 18 +++ .../kubernetes/KubernetesRuntimeFactoryTest.java | 10 ++-- .../runtime/kubernetes/KubernetesRuntimeTest.java | 6 +-- .../runtime/process/ProcessRuntimeTest.java| 6 +-- pulsar-functions/secrets/pom.xml | 2 +- .../DefaultSecretsProviderConfigurator.java| 2 +- .../KubernetesSecretsProviderConfigurator.java | 10 ++-- .../SecretsProviderConfigurator.java | 6 +-- .../pulsar/io/elasticsearch/ElasticSearchSink.java | 11 ++-- .../integration/io/ElasticSearchSinkTester.java| 5 +- 19 files changed, 113 insertions(+), 106 deletions(-)
[pulsar] branch master updated (6eefee7 -> 37a1d87)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 6eefee7 [Issue 7759] Support set Max Consumer on topic level. (#7968) add 37a1d87 Remove stateful functions note (#7956) No new revisions were added by this update. Summary of changes: site2/docs/deploy-bare-metal.md | 9 - .../versioned_docs/version-2.1.0-incubating/deploy-bare-metal.md | 9 - .../versioned_docs/version-2.1.1-incubating/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.2.0/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.2.1/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.3.0/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.3.1/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.3.2/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.4.0/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.4.1/develop-bare-metal.md | 9 - site2/website/versioned_docs/version-2.4.2/develop-bare-metal.md | 9 - site2/website/versioned_docs/version-2.5.0/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.5.1/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.5.2/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.6.0/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.6.1/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.6.2/deploy-bare-metal.md | 9 - 17 files changed, 153 deletions(-)
[pulsar] branch master updated (8f2540f -> 5adbc1f)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 8f2540f [Transaction] Txn ack abort process on subscription (#7979) add 5adbc1f [Security] Upgrade the snakeyaml verion to 1.26 (#7994) No new revisions were added by this update. Summary of changes: distribution/server/src/assemble/LICENSE.bin.txt | 23 ++--- pom.xml| 4 +- pulsar-functions/runtime/pom.xml | 2 +- .../auth/KubernetesFunctionAuthProvider.java | 4 +- .../auth/KubernetesSecretsTokenAuthProvider.java | 34 ++-- .../BasicKubernetesManifestCustomizer.java | 2 +- .../kubernetes/KubernetesManifestCustomizer.java | 4 +- .../runtime/kubernetes/KubernetesRuntime.java | 60 +++--- .../kubernetes/KubernetesRuntimeFactory.java | 10 ++-- .../KubernetesSecretsTokenAuthProviderTest.java| 18 +++ .../kubernetes/KubernetesRuntimeFactoryTest.java | 10 ++-- .../runtime/kubernetes/KubernetesRuntimeTest.java | 6 +-- .../runtime/process/ProcessRuntimeTest.java| 6 +-- pulsar-functions/secrets/pom.xml | 2 +- .../DefaultSecretsProviderConfigurator.java| 2 +- .../KubernetesSecretsProviderConfigurator.java | 10 ++-- .../SecretsProviderConfigurator.java | 6 +-- .../pulsar/io/elasticsearch/ElasticSearchSink.java | 11 ++-- .../integration/io/ElasticSearchSinkTester.java| 5 +- 19 files changed, 113 insertions(+), 106 deletions(-)
[pulsar] branch master updated (6eefee7 -> 37a1d87)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 6eefee7 [Issue 7759] Support set Max Consumer on topic level. (#7968) add 37a1d87 Remove stateful functions note (#7956) No new revisions were added by this update. Summary of changes: site2/docs/deploy-bare-metal.md | 9 - .../versioned_docs/version-2.1.0-incubating/deploy-bare-metal.md | 9 - .../versioned_docs/version-2.1.1-incubating/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.2.0/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.2.1/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.3.0/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.3.1/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.3.2/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.4.0/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.4.1/develop-bare-metal.md | 9 - site2/website/versioned_docs/version-2.4.2/develop-bare-metal.md | 9 - site2/website/versioned_docs/version-2.5.0/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.5.1/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.5.2/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.6.0/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.6.1/deploy-bare-metal.md | 9 - site2/website/versioned_docs/version-2.6.2/deploy-bare-metal.md | 9 - 17 files changed, 153 deletions(-)
[pulsar] branch master updated: Report compacted topic ledger info when calling get internal stats. (#7988)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 97ba09e Report compacted topic ledger info when calling get internal
stats. (#7988)
97ba09e is described below
commit 97ba09e63be256412ff013611dca1f000cff9b9a
Author: Marvin Cai
AuthorDate: Mon Sep 7 17:13:25 2020 -0700
Report compacted topic ledger info when calling get internal stats. (#7988)
Fixes #7895
### Motivation
For get-internal-stats of persistent topic admin cli:
https://pulsar.apache.org/docs/en/2.6.0/admin-api-persistent-topics/#get-internal-stats,
we can also return the compacted topic ledger id if compaction is enabled. So
we'll able to read from ledger without creating additional subscription, it can
benefit like querying compacted topic from Pulsar SQL.
### Modifications
Expose CompactedTopicContext from CompactedTopicImpl, try to get ledger
information of compacted topic ledger if exist in PersistentTopic.
### Verifying this change
This change added tests and can be verified as follows:
- Added unit test to verify correct compacted ledger info is returned after
compaction.
* Report compacted topic ledger info when calling get internal stats.
* Update documentation to add information about returning compacted topic
ledger when get-internal-stats.
---
.../broker/service/persistent/PersistentTopic.java | 21 +
.../pulsar/compaction/CompactedTopicImpl.java | 14 +-
.../apache/pulsar/compaction/CompactionTest.java| 7 +++
.../policies/data/PersistentTopicInternalStats.java | 3 +++
site2/docs/admin-api-persistent-topics.md | 19 ++-
5 files changed, 62 insertions(+), 2 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
index 8ff51f6..becd094 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java
@@ -38,6 +38,7 @@ import java.util.Set;
import java.util.concurrent.CancellationException;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionException;
+import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicLong;
@@ -1605,6 +1606,26 @@ public class PersistentTopic extends AbstractTopic
implements Topic, AddEntryCal
stats.ledgers.add(info);
});
+// Add ledger info for compacted topic ledger if exist.
+LedgerInfo info = new LedgerInfo();
+info.ledgerId = -1;
+info.entries = -1;
+info.size = -1;
+
+try {
+Optional
compactedTopicContext =
+
((CompactedTopicImpl)compactedTopic).getCompactedTopicContext();
+if (compactedTopicContext.isPresent()) {
+CompactedTopicImpl.CompactedTopicContext ledgerContext =
compactedTopicContext.get();
+info.ledgerId = ledgerContext.getLedger().getId();
+info.entries = ledgerContext.getLedger().getLastAddConfirmed()
+ 1;
+info.size = ledgerContext.getLedger().getLength();
+}
+} catch (ExecutionException | InterruptedException e) {
+log.warn("[{}]Fail to get ledger information for compacted
topic.", topic);
+}
+stats.compactedLedger = info;
+
stats.cursors = Maps.newTreeMap();
ml.getCursors().forEach(c -> {
ManagedCursorImpl cursor = (ManagedCursorImpl) c;
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java
b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java
index 2d430a7..74d24c6 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/compaction/CompactedTopicImpl.java
@@ -28,8 +28,11 @@ import java.util.Collections;
import java.util.Enumeration;
import java.util.List;
import java.util.NoSuchElementException;
+import java.util.Optional;
import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import lombok.Getter;
import org.apache.bookkeeper.client.BookKeeper;
import org.apache.bookkeeper.client.BKException;
import org.apache.bookkeeper.client.LedgerHandle;
@@ -250,7 +253,8 @@ public class CompactedTopicImpl implements CompactedTopic {
});
}
-static class Compacte
[pulsar] branch master updated (d45ee07 -> 7b5301d)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from d45ee07 Improve batch source intermediate topic cleanup (#7985) add 7b5301d Add bookkeeperClientMinNumRacksPerWriteQuorum, bookkeeperClientEnforceMinNumRacksPerWriteQuorum conf options (#7977) No new revisions were added by this update. Summary of changes: conf/broker.conf | 9 + conf/standalone.conf | 9 + deployment/terraform-ansible/templates/broker.conf | 9 + .../org/apache/pulsar/broker/ServiceConfiguration.java | 10 ++ .../pulsar/broker/BookKeeperClientFactoryImpl.java | 4 .../java/org/apache/pulsar/PulsarBrokerStarterTest.java | 4 .../pulsar/broker/BookKeeperClientFactoryImplTest.java | 16 .../resources/configurations/pulsar_broker_test.conf | 2 ++ pulsar-client-cpp/test-conf/standalone-ssl.conf | 9 + pulsar-client-cpp/test-conf/standalone.conf | 9 + pulsar-client-cpp/tests/authentication.conf | 9 + pulsar-client-cpp/tests/standalone.conf | 9 + site2/docs/reference-configuration.md| 4 .../version-2.6.2/reference-configuration.md | 2 ++ 14 files changed, 105 insertions(+)
[pulsar] branch master updated: [Admin cli]Fixed website admin cli link error (#7969)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 5432e86 [Admin cli]Fixed website admin cli link error (#7969)
5432e86 is described below
commit 5432e86a4ad4686cd308fd0b7d32d028b674380a
Author: Guangning
AuthorDate: Thu Sep 3 21:13:31 2020 +0800
[Admin cli]Fixed website admin cli link error (#7969)
Motivation
Website admin cli link error
Modifications
Update link version
Verifying this change
Local test
---
site2/website/static/js/getCliByVersion.js | 1 +
1 file changed, 1 insertion(+)
diff --git a/site2/website/static/js/getCliByVersion.js
b/site2/website/static/js/getCliByVersion.js
index f817a91..d2b6b97 100644
--- a/site2/website/static/js/getCliByVersion.js
+++ b/site2/website/static/js/getCliByVersion.js
@@ -27,6 +27,7 @@ function getCliByVersion(){
return
}
} else {
+version = parseInt(versions[0]) + "." + parseInt(versions[1]) + ".0"
window.location.href = "http://pulsar.apache.org/tools/pulsar-admin/;
+ version + "-SNAPSHOT"
return
}
[pulsar] branch master updated: [Doc]--Update BookKeeper config (#7834)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 9994d6d [Doc]--Update BookKeeper config (#7834) 9994d6d is described below commit 9994d6dc11a9c04a674d4fec1cc0dc19a6526b3b Author: HuanliMeng <48120384+huanli-m...@users.noreply.github.com> AuthorDate: Tue Sep 1 23:21:13 2020 +0800 [Doc]--Update BookKeeper config (#7834) Fixes #7481 Motivation Default bookkeeper conf that ships with apache pulsar indicates that the unit of openLedgerRereplicationGracePeriod is in seconds, but official bookkeeper doc(and code) mentions that the unit is in milliseconds. Pulsar should configure default as 3 instead of 30 and bookkeeper.conf should mention that the unit of this period is milliseconds instead of seconds. Modifications Update the bookkeep.config file Update the following Pulsar docs for releases 2.5.2, 2.6.0, 2.6.1, and master Reference > Pulsar Config > BookKeeper --- conf/bookkeeper.conf | 4 ++-- site2/docs/reference-configuration.md | 2 +- site2/website/versioned_docs/version-2.5.2/reference-configuration.md | 1 + site2/website/versioned_docs/version-2.6.0/reference-configuration.md | 1 + site2/website/versioned_docs/version-2.6.1/reference-configuration.md | 1 + 5 files changed, 6 insertions(+), 3 deletions(-) diff --git a/conf/bookkeeper.conf b/conf/bookkeeper.conf index eefdb83..2978e4c 100644 --- a/conf/bookkeeper.conf +++ b/conf/bookkeeper.conf @@ -227,9 +227,9 @@ auditorPeriodicBookieCheckInterval=86400 rereplicationEntryBatchSize=100 # Auto-replication -# The grace period, in seconds, that the replication worker waits before fencing and +# The grace period, in milliseconds, that the replication worker waits before fencing and # replicating a ledger fragment that's still being written to upon bookie failure. -openLedgerRereplicationGracePeriod=30 +openLedgerRereplicationGracePeriod=3 # Whether the bookie itself can start auto-recovery service also or not autoRecoveryDaemonEnabled=true diff --git a/site2/docs/reference-configuration.md b/site2/docs/reference-configuration.md index 80f9393..d37bc2d 100644 --- a/site2/docs/reference-configuration.md +++ b/site2/docs/reference-configuration.md @@ -68,8 +68,8 @@ BookKeeper is a replicated log storage system that Pulsar uses for durable stora |journalBufferedWritesThreshold|Maximum writes to buffer to achieve grouping|524288| |journalFlushWhenQueueEmpty|If we should flush the journal when journal queue is empty|false| |numJournalCallbackThreads|The number of threads that should handle journal callbacks|8| +|openLedgerRereplicationGracePeriod | The grace period, in milliseconds, that the replication worker waits before fencing and replicating a ledger fragment that's still being written to upon bookie failure. | 3 | |rereplicationEntryBatchSize|The number of max entries to keep in fragment for re-replication|100| -|openLedgerRereplicationGracePeriod|The grace period, in seconds, that the replication worker waits before fencing and replicating a ledger fragment that's still being written to upon the bookie failure.|30| |autoRecoveryDaemonEnabled|Whether the bookie itself can start auto-recovery service.|true| |lostBookieRecoveryDelay|How long to wait, in seconds, before starting auto recovery of a lost bookie.|0| |gcWaitTime|How long the interval to trigger next garbage collection, in milliseconds. Since garbage collection is running in background, too frequent gc will heart performance. It is better to give a higher number of gc interval if there is enough disk capacity.|90| diff --git a/site2/website/versioned_docs/version-2.5.2/reference-configuration.md b/site2/website/versioned_docs/version-2.5.2/reference-configuration.md index a3ea91a..362988c 100644 --- a/site2/website/versioned_docs/version-2.5.2/reference-configuration.md +++ b/site2/website/versioned_docs/version-2.5.2/reference-configuration.md @@ -63,6 +63,7 @@ BookKeeper is a replicated log storage system that Pulsar uses for durable stora |journalFlushWhenQueueEmpty|If we should flush the journal when journal queue is empty|false| |numJournalCallbackThreads|The number of threads that should handle journal callbacks|8| |rereplicationEntryBatchSize|The number of max entries to keep in fragment for re-replication|5000| +| openLedgerRereplicationGracePeriod | The grace period, in milliseconds, that the replication worker waits before fencing and replicating a ledger fragment that's still being written to upon bookie failure. | 3 | |gcWaitTime|How long the interval to trigger next garbage collection, in milliseconds. Since garbage collection is running in background, too
[pulsar] branch master updated: support use `BitSet` generate the `BatchMessageAcker` (#7909)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 81202e1 support use `BitSet` generate the `BatchMessageAcker` (#7909)
81202e1 is described below
commit 81202e147f9460e3ab60d82fe0ad55510a258ed6
Author: ran
AuthorDate: Tue Sep 1 11:54:21 2020 +0800
support use `BitSet` generate the `BatchMessageAcker` (#7909)
Motivation
Currently, we have to know the batchSize to generate BatchMessageAcker. If
we could get the batch index ack bitSet from Broker we could generate the
BatchMessageAcker by the bitSet, this is useful for consuming transaction
messages, we don't need to change the protocol to get the total message number
of one transaction.
Modifications
Add a new static method to generate the BatchMessageAcker by BitSet.
---
.../pulsar/client/impl/BatchMessageAcker.java | 5 +
.../apache/pulsar/client/impl/ConsumerImpl.java| 9 +++--
.../PersistentAcknowledgmentsGroupingTracker.java | 18 +
.../pulsar/client/impl/BatchMessageAckerTest.java | 12 +++
.../collections/ConcurrentBitSetRecyclable.java| 7 +++
.../ConcurrentBitSetRecyclableTest.java| 23 ++
6 files changed, 68 insertions(+), 6 deletions(-)
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageAcker.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageAcker.java
index d46d3b3..e34d1a1 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageAcker.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageAcker.java
@@ -34,6 +34,11 @@ class BatchMessageAcker {
return new BatchMessageAcker(bitSet, batchSize);
}
+// Use the param bitSet as the BatchMessageAcker's bitSet, don't care
about the batchSize.
+static BatchMessageAcker newAcker(BitSet bitSet) {
+return new BatchMessageAcker(bitSet, -1);
+}
+
// bitset shared across messages in the same batch.
private final int batchSize;
private final BitSet bitSet;
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
index b1df6f8..cfaaa89 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
@@ -1356,16 +1356,21 @@ public class ConsumerImpl extends ConsumerBase
implements ConnectionHandle
// create ack tracker for entry aka batch
MessageIdImpl batchMessage = new
MessageIdImpl(messageId.getLedgerId(), messageId.getEntryId(),
getPartitionIndex());
-BatchMessageAcker acker = BatchMessageAcker.newAcker(batchSize);
List> possibleToDeadLetter = null;
if (deadLetterPolicy != null && redeliveryCount >=
deadLetterPolicy.getMaxRedeliverCount()) {
possibleToDeadLetter = new ArrayList<>();
}
-int skippedMessages = 0;
+
+BatchMessageAcker acker;
BitSetRecyclable ackBitSet = null;
if (ackSet != null && ackSet.size() > 0) {
ackBitSet =
BitSetRecyclable.valueOf(SafeCollectionUtils.longListToArray(ackSet));
+acker =
BatchMessageAcker.newAcker(BitSet.valueOf(SafeCollectionUtils.longListToArray(ackSet)));
+} else {
+acker = BatchMessageAcker.newAcker(batchSize);
}
+
+int skippedMessages = 0;
try {
int startBatchIndex = Math.max(messageId.getBatchIndex(), 0);
for (int i = startBatchIndex; i < batchSize; ++i) {
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java
index 6a4deef..fd61c42 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java
@@ -168,8 +168,13 @@ public class PersistentAcknowledgmentsGroupingTracker
implements Acknowledgments
} else if (ackType == AckType.Individual) {
ConcurrentBitSetRecyclable bitSet =
pendingIndividualBatchIndexAcks.computeIfAbsent(
new MessageIdImpl(msgId.getLedgerId(), msgId.getEntryId(),
msgId.getPartitionIndex()), (v) -> {
-ConcurrentBitSetRecyclable value =
ConcurrentBitSetRecyclable.create();
-value.set(0, batchSize);
+ConcurrentBitSetRecyclable value;
+if (msgId.g
[pulsar] branch master updated: Update reference-configuration.md (#7924)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new d3dd31f Update reference-configuration.md (#7924) d3dd31f is described below commit d3dd31f8633dc5ff72870c9a8a591b398444ee04 Author: sijia-w <53718687+siji...@users.noreply.github.com> AuthorDate: Sat Aug 29 09:54:33 2020 +0200 Update reference-configuration.md (#7924) Update AWS deployment in reference-configuration. --- site2/docs/reference-configuration.md | 169 +++--- 1 file changed, 135 insertions(+), 34 deletions(-) diff --git a/site2/docs/reference-configuration.md b/site2/docs/reference-configuration.md index 5664db7..80f9393 100644 --- a/site2/docs/reference-configuration.md +++ b/site2/docs/reference-configuration.md @@ -34,7 +34,11 @@ BookKeeper is a replicated log storage system that Pulsar uses for durable stora |bookiePort|The port on which the bookie server listens.|3181| |allowLoopback|Whether the bookie is allowed to use a loopback interface as its primary interface (i.e. the interface used to establish its identity). By default, loopback interfaces are not allowed as the primary interface. Using a loopback interface as the primary interface usually indicates a configuration error. For example, it’s fairly common in some VPS setups to not configure a hostname or to have the hostname resolve to `127.0.0.1`. If this is the case, then all bookies in the cl [...] |listeningInterface|The network interface on which the bookie listens. If not set, the bookie will listen on all interfaces.|eth0| +|advertisedAddress|Configure a specific hostname or IP address that the bookie should use to advertise itself to clients. If not set, bookie will advertised its own IP address or hostname, depending on the `listeningInterface` and `useHostNameAsBookieID` settings.|N/A| +|allowMultipleDirsUnderSameDiskPartition|Configure the bookie to allow/disallow multiple ledger/index/journal directories in the same filesystem disk partition|false| +|minUsableSizeForIndexFileCreation|The minimum safe usable size available in index directory for bookie to create index files while replaying journal at the time of bookie starts in Readonly Mode (in bytes).|1073741824| |journalDirectory|The directory where Bookkeeper outputs its write-ahead log (WAL)|data/bookkeeper/journal| +|journalDirectories|Directories that BookKeeper outputs its write ahead log. Multi directories are available, being separated by `,`. For example: `journalDirectories=/tmp/bk-journal1,/tmp/bk-journal2`. If `journalDirectories` is set, bookies will skip `journalDirectory` and use this setting directory.|/tmp/bk-journal| |ledgerDirectories|The directory where Bookkeeper outputs ledger snapshots. This could define multiple directories to store snapshots separated by comma, for example `ledgerDirectories=/tmp/bk1-data,/tmp/bk2-data`. Ideally, ledger dirs and the journal dir are each in a different device, which reduces the contention between random I/O and sequential write. It is possible to run with a single disk, but performance will be significantly lower.|data/bookkeeper/ledgers| |ledgerManagerType|The type of ledger manager used to manage how ledgers are stored, managed, and garbage collected. See [BookKeeper Internals](http://bookkeeper.apache.org/docs/latest/getting-started/concepts) for more info.|hierarchical| |zkLedgersRootPath|The root ZooKeeper path used to store ledger metadata. This parameter is used by the ZooKeeper-based ledger manager as a root znode to store all ledgers.|/ledgers| @@ -42,9 +46,12 @@ BookKeeper is a replicated log storage system that Pulsar uses for durable stora |entryLogFilePreallocationEnabled|Enable or disable entry logger preallocation|true| |logSizeLimit|Max file size of the entry logger, in bytes. A new entry log file will be created when the old one reaches the file size limitation.|2147483648| |minorCompactionThreshold|Threshold of minor compaction. Entry log files whose remaining size percentage reaches below this threshold will be compacted in a minor compaction. If set to less than zero, the minor compaction is disabled.|0.2| -|minorCompactionInterval|Time interval to run minor compaction, in seconds. If set to less than zero, the minor compaction is disabled.|3600| +|minorCompactionInterval|Time interval to run minor compaction, in seconds. If set to less than zero, the minor compaction is disabled. Note: should be greater than gcWaitTime. |3600| |majorCompactionThreshold|The threshold of major compaction. Entry log files whose remaining size percentage reaches below this threshold will be compacted in a major compaction. Those entry log files whose remaining size percentage is still higher than the threshold will never be compacted. If set to less than
svn commit: r41134 - /dev/pulsar/pulsar-client-go-0.2.0-candidate-1/ /release/pulsar/pulsar-client-go-0.2.0/
Author: zhaijia Date: Thu Aug 27 02:23:30 2020 New Revision: 41134 Log: release 0.2.0 Added: release/pulsar/pulsar-client-go-0.2.0/ - copied from r41133, dev/pulsar/pulsar-client-go-0.2.0-candidate-1/ Removed: dev/pulsar/pulsar-client-go-0.2.0-candidate-1/
[pulsar] branch branch-2.6 updated: Upgrade Presto version to 332 (#7194)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch branch-2.6
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/branch-2.6 by this push:
new 2e44cb1 Upgrade Presto version to 332 (#7194)
2e44cb1 is described below
commit 2e44cb15c77dc24aa555c720a16df777a5facfa0
Author: Yuya Ebihara
AuthorDate: Sun Jun 14 10:14:19 2020 +0900
Upgrade Presto version to 332 (#7194)
* Upgrade Presto version to 332
* Update LICENSE file in pulsar-sql module
* Wrap ClassLoaderSafeConnectorMetadata around PulsarMetadata
* Update LICENSE file in pulsar-sql module
(cherry picked from commit 5d02c70b674a36e5bd36c60e488737db5740d21e)
---
bin/pulsar | 2 +-
conf/presto/log.properties | 4 +-
pom.xml| 4 +-
pulsar-sql/presto-distribution/LICENSE | 148 -
pulsar-sql/presto-distribution/pom.xml | 8 +-
pulsar-sql/presto-pulsar/pom.xml | 10 +-
.../pulsar/sql/presto/JSONSchemaHandler.java | 2 +-
.../pulsar/sql/presto/PulsarColumnHandle.java | 6 +-
.../pulsar/sql/presto/PulsarColumnMetadata.java| 4 +-
.../apache/pulsar/sql/presto/PulsarConnector.java | 19 +--
.../pulsar/sql/presto/PulsarConnectorFactory.java | 8 +-
.../pulsar/sql/presto/PulsarConnectorModule.java | 8 +-
.../pulsar/sql/presto/PulsarHandleResolver.java| 12 +-
.../pulsar/sql/presto/PulsarInternalColumn.java| 10 +-
.../apache/pulsar/sql/presto/PulsarMetadata.java | 79 +--
.../org/apache/pulsar/sql/presto/PulsarPlugin.java | 4 +-
.../pulsar/sql/presto/PulsarRecordCursor.java | 32 ++---
.../apache/pulsar/sql/presto/PulsarRecordSet.java | 6 +-
.../pulsar/sql/presto/PulsarRecordSetProvider.java | 12 +-
.../pulsar/sql/presto/PulsarSchemaHandlers.java| 7 +-
.../org/apache/pulsar/sql/presto/PulsarSplit.java | 8 +-
.../pulsar/sql/presto/PulsarSplitManager.java | 26 ++--
.../pulsar/sql/presto/PulsarTableHandle.java | 4 +-
.../pulsar/sql/presto/PulsarTableLayoutHandle.java | 6 +-
.../pulsar/sql/presto/PulsarTransactionHandle.java | 2 +-
...k.presto.spi.Plugin => io.prestosql.spi.Plugin} | 0
.../pulsar/sql/presto/TestAvroSchemaHandler.java | 4 +-
.../pulsar/sql/presto/TestPulsarConnector.java | 22 +--
.../presto/TestPulsarKeyValueSchemaHandler.java| 2 +-
.../pulsar/sql/presto/TestPulsarMetadata.java | 31 ++---
.../presto/TestPulsarPrimitiveSchemaHandler.java | 4 +-
.../pulsar/sql/presto/TestPulsarSplitManager.java | 20 +--
site2/docs/sql-deployment-configurations.md| 6 +-
.../version-2.2.0/sql-deployment-configurations.md | 4 +-
.../versioned_docs/version-2.2.0/sql-overview.md | 2 +-
.../version-2.5.0/sql-deployment-configurations.md | 4 +-
tests/integration/pom.xml | 2 +-
37 files changed, 281 insertions(+), 251 deletions(-)
diff --git a/bin/pulsar b/bin/pulsar
index ffcdf53..7522df9 100755
--- a/bin/pulsar
+++ b/bin/pulsar
@@ -348,7 +348,7 @@ elif [ $COMMAND == "compact-topic" ]; then
exec $JAVA $OPTS org.apache.pulsar.compaction.CompactorTool --broker-conf
$PULSAR_BROKER_CONF $@
elif [ $COMMAND == "sql" ]; then
check_presto_libraries
-exec $JAVA -cp "${PRESTO_HOME}/lib/*" com.facebook.presto.cli.Presto
--server localhost:8081 "${@}"
+exec $JAVA -cp "${PRESTO_HOME}/lib/*" io.prestosql.cli.Presto --server
localhost:8081 "${@}"
elif [ $COMMAND == "sql-worker" ]; then
check_presto_libraries
exec ${PRESTO_HOME}/bin/launcher --etc-dir ${PULSAR_PRESTO_CONF} "${@}"
diff --git a/conf/presto/log.properties b/conf/presto/log.properties
index 5f02636..000961f 100644
--- a/conf/presto/log.properties
+++ b/conf/presto/log.properties
@@ -17,7 +17,7 @@
# under the License.
#
-com.facebook.presto=INFO
+io.prestosql=INFO
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory=WARN
com.ning.http.client=WARN
-com.facebook.presto.server.PluginManager=DEBUG
+io.prestosql.server.PluginManager=DEBUG
diff --git a/pom.xml b/pom.xml
index f8c13b2..167aa3e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -203,7 +203,7 @@ flexible messaging model and an intuitive client
API.
9.3.24.v20180605
9.3.24.v20180605
6.3.2
-0.206
+332
1.6.0
2.11
1.0.0.Final
@@ -1447,7 +1447,7 @@ flexible messaging model and an intuitive client
API.
**/META-INF/services/com.scurrilous.circe.HashProvider
-
**/META-INF/services/com.facebook.presto.spi.Plugin
+**/META-INF/services/io.prestosql.spi.Plugin
**/django/stats/migrations/*.py
diff --git a/pulsar-sql/presto-di
[pulsar] branch master updated: [issue 7851][C++] Make clear() thread-safe (#7862)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 97f4112 [issue 7851][C++] Make clear() thread-safe (#7862)
97f4112 is described below
commit 97f41120b9691474f0038b220f3204fa69e48257
Author: Yunze Xu
AuthorDate: Mon Aug 24 07:39:32 2020 +0800
[issue 7851][C++] Make clear() thread-safe (#7862)
Fixes #7851
### Motivation
`clear()` methods of `BatchAcknowledgementTracker` and
`UnAckedMessageTrackerEnabled` are not thread-safe.
### Modifications
Acquire a mutex in these `clear()` methods.
---
pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc | 1 +
pulsar-client-cpp/lib/UnAckedMessageTrackerEnabled.cc | 6 +-
pulsar-client-cpp/tests/BasicEndToEndTest.cc | 2 ++
3 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc
b/pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc
index df15119..3d6d920 100644
--- a/pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc
+++ b/pulsar-client-cpp/lib/BatchAcknowledgementTracker.cc
@@ -33,6 +33,7 @@
BatchAcknowledgementTracker::BatchAcknowledgementTracker(const std::string topic
}
void BatchAcknowledgementTracker::clear() {
+Lock lock(mutex_);
trackerMap_.clear();
sendList_.clear();
}
diff --git a/pulsar-client-cpp/lib/UnAckedMessageTrackerEnabled.cc
b/pulsar-client-cpp/lib/UnAckedMessageTrackerEnabled.cc
index 9185dba..e280dba 100644
--- a/pulsar-client-cpp/lib/UnAckedMessageTrackerEnabled.cc
+++ b/pulsar-client-cpp/lib/UnAckedMessageTrackerEnabled.cc
@@ -39,7 +39,7 @@ void UnAckedMessageTrackerEnabled::timeoutHandler() {
}
void UnAckedMessageTrackerEnabled::timeoutHandlerHelper() {
-std::lock_guard acquire(lock_);
+std::unique_lock acquire(lock_);
LOG_DEBUG("UnAckedMessageTrackerEnabled::timeoutHandlerHelper invoked for
consumerPtr_ "
<< consumerReference_.getName().c_str());
@@ -60,6 +60,9 @@ void UnAckedMessageTrackerEnabled::timeoutHandlerHelper() {
timePartitions.push_back(headPartition);
if (msgIdsToRedeliver.size() > 0) {
+// redeliverUnacknowledgedMessages() may call clear() that acquire the
lock again, so we should unlock
+// here to avoid deadlock
+acquire.unlock();
consumerReference_.redeliverUnacknowledgedMessages(msgIdsToRedeliver);
}
}
@@ -148,6 +151,7 @@ void UnAckedMessageTrackerEnabled::removeTopicMessage(const
std::string& topic)
}
void UnAckedMessageTrackerEnabled::clear() {
+std::lock_guard acquire(lock_);
messageIdPartitionMap.clear();
for (auto it = timePartitions.begin(); it != timePartitions.end(); it++) {
it->clear();
diff --git a/pulsar-client-cpp/tests/BasicEndToEndTest.cc
b/pulsar-client-cpp/tests/BasicEndToEndTest.cc
index b574630..eeb1489 100644
--- a/pulsar-client-cpp/tests/BasicEndToEndTest.cc
+++ b/pulsar-client-cpp/tests/BasicEndToEndTest.cc
@@ -1750,6 +1750,8 @@ TEST(BasicEndToEndTest,
testPartitionTopicUnAckedMessageTimeout) {
std::this_thread::sleep_for(std::chrono::milliseconds(500));
timeWaited += 500;
}
+
+client.close();
}
TEST(BasicEndToEndTest, testUnAckedMessageTimeoutListener) {
[pulsar-helm-chart] branch master updated: Bump the image version to 2.6.1 (#57)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar-helm-chart.git The following commit(s) were added to refs/heads/master by this push: new 4178c70 Bump the image version to 2.6.1 (#57) 4178c70 is described below commit 4178c70d9093325b29a40441f613919c1b014ce8 Author: 冉小龙 AuthorDate: Fri Aug 21 22:50:27 2020 +0800 Bump the image version to 2.6.1 (#57) Signed-off-by: xiaolong.ran r...@apache.org Motivation Follow release process and bump the image version to 2.6.1 --- charts/pulsar/Chart.yaml | 4 ++-- charts/pulsar/values.yaml | 14 +++--- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/charts/pulsar/Chart.yaml b/charts/pulsar/Chart.yaml index 3e1577a..e47655a 100644 --- a/charts/pulsar/Chart.yaml +++ b/charts/pulsar/Chart.yaml @@ -18,10 +18,10 @@ # apiVersion: v1 -appVersion: "2.6.0" +appVersion: "2.6.1" description: Apache Pulsar Helm chart for Kubernetes name: pulsar -version: 2.6.0-3 +version: 2.6.1-1 home: https://pulsar.apache.org sources: - https://github.com/apache/pulsar diff --git a/charts/pulsar/values.yaml b/charts/pulsar/values.yaml index df9d3cd..18d5d7c 100644 --- a/charts/pulsar/values.yaml +++ b/charts/pulsar/values.yaml @@ -131,27 +131,27 @@ extra: images: zookeeper: repository: apachepulsar/pulsar-all -tag: 2.6.0 +tag: 2.6.1 pullPolicy: IfNotPresent bookie: repository: apachepulsar/pulsar-all -tag: 2.6.0 +tag: 2.6.1 pullPolicy: IfNotPresent autorecovery: repository: apachepulsar/pulsar-all -tag: 2.6.0 +tag: 2.6.1 pullPolicy: IfNotPresent broker: repository: apachepulsar/pulsar-all -tag: 2.6.0 +tag: 2.6.1 pullPolicy: IfNotPresent proxy: repository: apachepulsar/pulsar-all -tag: 2.6.0 +tag: 2.6.1 pullPolicy: IfNotPresent functions: repository: apachepulsar/pulsar-all -tag: 2.6.0 +tag: 2.6.1 prometheus: repository: prom/prometheus tag: v2.17.2 @@ -536,7 +536,7 @@ pulsar_metadata: image: # the image used for running `pulsar-cluster-initialize` job repository: apachepulsar/pulsar-all -tag: 2.6.0 +tag: 2.6.1 pullPolicy: IfNotPresent ## set an existing configuration store # configurationStore:
[pulsar] branch master updated (cdce332 -> cc89e0c)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from cdce332 Remove the unused CI job (#7853) add cc89e0c Some clarification of docs for partition and ordering key (#7745) No new revisions were added by this update. Summary of changes: pulsar-client-cpp/include/pulsar/MessageBuilder.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[pulsar] branch master updated (7d9319d -> 2ee109e)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from 7d9319d [Client]Add autoPartitionsUpdateInterval for producer and consumer (#7840) add 2ee109e Fix: close() never gets called for BatchSource (#7866) No new revisions were added by this update. Summary of changes: .../pulsar/functions/source/batch/BatchSourceExecutor.java | 3 +++ .../functions/source/batch/BatchSourceExecutorTest.java | 12 ++-- 2 files changed, 13 insertions(+), 2 deletions(-)
