[GitHub] hrsakai opened a new pull request #1986: Fixed message popover def
hrsakai opened a new pull request #1986: Fixed message popover def URL: https://github.com/apache/incubator-pulsar/pull/1986 ### Motivation Currently, site can't display message popover correctly. ### Modifications Remove double quote from message def in popovers.yaml. ### Result For example, in [pulsar functions overview page](http://hsakai.pulsar.ssk.ynwm.yahoo.co.jp:4000/docs/latest/functions/overview/): **before** https://user-images.githubusercontent.com/22829228/41578353-cdc928d4-73cc-11e8-8c07-e284cf4f8be8.png; width=400> **after** https://user-images.githubusercontent.com/22829228/41578379-f2cd0be6-73cc-11e8-94b9-102a0d650e02.png; width=400> This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] 01/01: Fixed python extension build
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch fix-python-build
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
commit 2f2104b0139f86ee765c71ea1ba889a062a94efc
Author: Matteo Merli
AuthorDate: Mon Jun 18 17:03:22 2018 -0700
Fixed python extension build
---
pulsar-client-cpp/docker/build-wheel-file-within-docker.sh | 14 +-
pulsar-client-cpp/docker/build-wheels.sh | 2 +-
pulsar-client-cpp/python/pulsar/__init__.py| 6 +++---
pulsar-client-cpp/python/setup.py | 2 +-
4 files changed, 14 insertions(+), 10 deletions(-)
diff --git a/pulsar-client-cpp/docker/build-wheel-file-within-docker.sh
b/pulsar-client-cpp/docker/build-wheel-file-within-docker.sh
index d550ca9..d4cb9af 100755
--- a/pulsar-client-cpp/docker/build-wheel-file-within-docker.sh
+++ b/pulsar-client-cpp/docker/build-wheel-file-within-docker.sh
@@ -1,4 +1,5 @@
#!/bin/bash
+
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
@@ -18,19 +19,22 @@
# under the License.
#
-set -e
+set -e -x
cd /pulsar/pulsar-client-cpp
-find . -name CMakeCache.txt | xargs rm
-find . -name CMakeFiles | xargs rm -rf
+find . -name CMakeCache.txt | xargs -r rm
+find . -name CMakeFiles | xargs -r rm -rf
cmake .
-DPYTHON_INCLUDE_DIR=/opt/python/$PYTHON_SPEC/include/python$PYTHON_VERSION \
-DPYTHON_LIBRARY=/opt/python/$PYTHON_SPEC/lib \
--DLINK_STATIC=ON
+-DLINK_STATIC=ON \
+-DBUILD_TESTS=OFF
make clean
-make _pulsar
+make _pulsar -j3
cd python
python setup.py bdist_wheel
+
+auditwheel repair dist/pulsar_client-*-$PYTHON_SPEC-linux_x86_64.whl
diff --git a/pulsar-client-cpp/docker/build-wheels.sh
b/pulsar-client-cpp/docker/build-wheels.sh
index 35cb789..2146869 100755
--- a/pulsar-client-cpp/docker/build-wheels.sh
+++ b/pulsar-client-cpp/docker/build-wheels.sh
@@ -44,5 +44,5 @@ for line in "${PYTHON_VERSIONS[@]}"; do
IMAGE_NAME=$BUILD_IMAGE_NAME:manylinux-$PYTHON_SPEC
echo "Using image: $IMAGE_NAME"
-docker run -i -v $PWD:/pulsar $IMAGE_NAME
/build-wheel-file-within-docker.sh
+docker run -i -v $PWD:/pulsar $IMAGE_NAME
/pulsar/pulsar-client-cpp/docker/build-wheel-file-within-docker.sh
done
diff --git a/pulsar-client-cpp/python/pulsar/__init__.py
b/pulsar-client-cpp/python/pulsar/__init__.py
index d8254d3..01a160f 100644
--- a/pulsar-client-cpp/python/pulsar/__init__.py
+++ b/pulsar-client-cpp/python/pulsar/__init__.py
@@ -100,9 +100,9 @@ import _pulsar
from _pulsar import Result, CompressionType, ConsumerType,
PartitionsRoutingMode # noqa: F401
-from functions.function import Function
-from functions.context import Context
-from functions.serde import SerDe, IdentitySerDe, PickleSerDe
+from pulsar.functions.function import Function
+from pulsar.functions.context import Context
+from pulsar.functions.serde import SerDe, IdentitySerDe, PickleSerDe
class MessageId:
"""
diff --git a/pulsar-client-cpp/python/setup.py
b/pulsar-client-cpp/python/setup.py
index ab91f0f..b20f3d6 100644
--- a/pulsar-client-cpp/python/setup.py
+++ b/pulsar-client-cpp/python/setup.py
@@ -60,7 +60,7 @@ class my_build_ext(build_ext.build_ext):
setup(
name="pulsar-client",
version=VERSION,
-py_modules=['pulsar'],
+packages=['pulsar', 'pulsar.functions'],
cmdclass={'build_ext': my_build_ext},
ext_modules=[Extension('_pulsar', [])],
[incubator-pulsar] branch fix-python-build updated (f0bd71a -> 2f2104b)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a change to branch fix-python-build in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git. discard f0bd71a Fixed python extension build new 2f2104b Fixed python extension build This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (f0bd71a) \ N -- N -- N refs/heads/fix-python-build (2f2104b) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: pulsar-client-cpp/python/pulsar/__init__.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
[GitHub] hrsakai closed issue #1929: [doc] Docker example and text out of sync
hrsakai closed issue #1929: [doc] Docker example and text out of sync URL: https://github.com/apache/incubator-pulsar/issues/1929 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Remove `--advertised-address` from docker getting-started section (#1984)
This is an automated email from the ASF dual-hosted git repository. hrsakai pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 37ce4fe Remove `--advertised-address` from docker getting-started section (#1984) 37ce4fe is described below commit 37ce4fe367ecb1969ddff48ba3d05295f92a7c67 Author: Sijie Guo AuthorDate: Mon Jun 18 20:52:11 2018 -0700 Remove `--advertised-address` from docker getting-started section (#1984) *Motivation* Fixes #1929 *Changes* Per [comment](https://github.com/apache/incubator-pulsar/issues/1929#issuecomment-395497432), we don't need to specify `--advertised-address` after 1.22.0 --- site/docs/latest/getting-started/docker.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/site/docs/latest/getting-started/docker.md b/site/docs/latest/getting-started/docker.md index 3ccd6f2..526494e 100644 --- a/site/docs/latest/getting-started/docker.md +++ b/site/docs/latest/getting-started/docker.md @@ -50,9 +50,6 @@ A few things to note about this command: * `-v $PWD/data:/pulsar/data`: This will make the process inside the container to store the data and metadata in the filesystem outside the container, in order to not start "fresh" every time the container is restarted. - * `--advertised-address 127.0.0.1`: This is needed so that the Pulsar broker can advertise an IP - address that is reachable from outside the Docker container. You can also use the host machine IP, - if you want to make Pulsar standalone accessible from other machines. If Pulsar has been successfully started, you should see `INFO`-level log messages like this:
[GitHub] hrsakai closed pull request #1984: Remove `--advertised-address` from docker getting-started section
hrsakai closed pull request #1984: Remove `--advertised-address` from docker getting-started section URL: https://github.com/apache/incubator-pulsar/pull/1984 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/site/docs/latest/getting-started/docker.md b/site/docs/latest/getting-started/docker.md index 3ccd6f2f3b..526494e7cc 100644 --- a/site/docs/latest/getting-started/docker.md +++ b/site/docs/latest/getting-started/docker.md @@ -50,9 +50,6 @@ A few things to note about this command: * `-v $PWD/data:/pulsar/data`: This will make the process inside the container to store the data and metadata in the filesystem outside the container, in order to not start "fresh" every time the container is restarted. - * `--advertised-address 127.0.0.1`: This is needed so that the Pulsar broker can advertise an IP - address that is reachable from outside the Docker container. You can also use the host machine IP, - if you want to make Pulsar standalone accessible from other machines. If Pulsar has been successfully started, you should see `INFO`-level log messages like this: This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat closed pull request #1985: Added release notes for 2.0.1-incubating release
merlimat closed pull request #1985: Added release notes for 2.0.1-incubating release URL: https://github.com/apache/incubator-pulsar/pull/1985 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/site/_config.yml b/site/_config.yml index ee337e357e..dd18a63c26 100644 --- a/site/_config.yml +++ b/site/_config.yml @@ -26,10 +26,11 @@ baseurl: / destination: ../generated-site/content preview_version_id: 20180426.125800-32 -current_version: 2.0.0-rc1-incubating -python_latest: 2.0.0 +current_version: 2.0.1-incubating +python_latest: 2.0.1 archived_releases: - 1.22.1-incubating + - 2.0.0-rc1-incubating - 1.22.0-incubating - 1.21.0-incubating - 1.20.0-incubating diff --git a/site/release-notes.md b/site/release-notes.md index 2c7aff36eb..48c6467af3 100644 --- a/site/release-notes.md +++ b/site/release-notes.md @@ -26,6 +26,42 @@ layout: content ## Apache incubator +### 2.0.1-incubating 2018-06-18 + +This release fixes issues reported for 2.0.0-rc1-incubating. + + * [#1893](https://github.com/apache/incubator-pulsar/pull/1893) - Fixed issues with Python packages on PyPI + * [#1797](https://github.com/apache/incubator-pulsar/issues/1797) - Proxy doesn't strip the request +URL for admin requests correctly + * [#1862](https://github.com/apache/incubator-pulsar/pull/1862) - Fix REST APIs provided by Pulsar proxy + +The complete list of changes can be found at: +https://github.com/apache/incubator-pulsar/milestone/14?closed=1 + +https://github.com/apache/incubator-pulsar/releases/tag/v2.0.1-incubating + +### 1.22.1-incubating 2018-06-18 + +This is the sixth release of Apache Pulsar since entering the ASF incubator. + +This release addresses issues reported in 1.22.0-incubating version. + + * [#1660](https://github.com/apache/incubator-pulsar/pull/1660) Deadlock while closing non persistent topic + * [#1591](https://github.com/apache/incubator-pulsar/pull/1591) Deadlock while closing non shared consumer + * [#1554](https://github.com/apache/incubator-pulsar/pull/1554) Handle invalid mark delete position at managed cursor + * [#1262](https://github.com/apache/incubator-pulsar/pull/1262) Broker should not start replicator for root partitioned topic + * [#1662](https://github.com/apache/incubator-pulsar/pull/1662) NPE when cursor failed to close empty subscription + * [#1370](https://github.com/apache/incubator-pulsar/pull/1370) Relocate service files for shading pulsar-client-admin module + * [#1265](https://github.com/apache/incubator-pulsar/pull/1265) Fixed lookup redirect logic on Proxyside + * [#1428](https://github.com/apache/incubator-pulsar/pull/1428) Handle Race condition in concurrent bundle split + * [#1817](https://github.com/apache/incubator-pulsar/pull/1817) Fixed mem leak when acknowledging while disconnected from broke + * [#1851](https://github.com/apache/incubator-pulsar/pull/1851) Fixing resource leak due to open file descriptors in SecurityUtility. + +The complete list of changes can be found at: +https://github.com/apache/incubator-pulsar/milestone/15?closed=1 + +https://github.com/apache/incubator-pulsar/releases/tag/v1.22.1-incubating + ### 2.0.0-rc1-incubating 2018-05-29 This is the fifth release of Apache Pulsar since entering the ASF incubator and the This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Added release notes for 2.0.1-incubating release (#1985)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 70412a9 Added release notes for 2.0.1-incubating release (#1985) 70412a9 is described below commit 70412a9c3c850bc76b93dc7882a3eb6f2713abee Author: Matteo Merli AuthorDate: Mon Jun 18 20:43:04 2018 -0700 Added release notes for 2.0.1-incubating release (#1985) * Added release notes for 2.0.1-incubating release` * Fixed tag link --- site/_config.yml | 5 +++-- site/release-notes.md | 36 2 files changed, 39 insertions(+), 2 deletions(-) diff --git a/site/_config.yml b/site/_config.yml index ee337e3..dd18a63 100644 --- a/site/_config.yml +++ b/site/_config.yml @@ -26,10 +26,11 @@ baseurl: / destination: ../generated-site/content preview_version_id: 20180426.125800-32 -current_version: 2.0.0-rc1-incubating -python_latest: 2.0.0 +current_version: 2.0.1-incubating +python_latest: 2.0.1 archived_releases: - 1.22.1-incubating + - 2.0.0-rc1-incubating - 1.22.0-incubating - 1.21.0-incubating - 1.20.0-incubating diff --git a/site/release-notes.md b/site/release-notes.md index 2c7aff3..48c6467 100644 --- a/site/release-notes.md +++ b/site/release-notes.md @@ -26,6 +26,42 @@ layout: content ## Apache incubator +### 2.0.1-incubating 2018-06-18 + +This release fixes issues reported for 2.0.0-rc1-incubating. + + * [#1893](https://github.com/apache/incubator-pulsar/pull/1893) - Fixed issues with Python packages on PyPI + * [#1797](https://github.com/apache/incubator-pulsar/issues/1797) - Proxy doesn't strip the request +URL for admin requests correctly + * [#1862](https://github.com/apache/incubator-pulsar/pull/1862) - Fix REST APIs provided by Pulsar proxy + +The complete list of changes can be found at: +https://github.com/apache/incubator-pulsar/milestone/14?closed=1 + +https://github.com/apache/incubator-pulsar/releases/tag/v2.0.1-incubating + +### 1.22.1-incubating 2018-06-18 + +This is the sixth release of Apache Pulsar since entering the ASF incubator. + +This release addresses issues reported in 1.22.0-incubating version. + + * [#1660](https://github.com/apache/incubator-pulsar/pull/1660) Deadlock while closing non persistent topic + * [#1591](https://github.com/apache/incubator-pulsar/pull/1591) Deadlock while closing non shared consumer + * [#1554](https://github.com/apache/incubator-pulsar/pull/1554) Handle invalid mark delete position at managed cursor + * [#1262](https://github.com/apache/incubator-pulsar/pull/1262) Broker should not start replicator for root partitioned topic + * [#1662](https://github.com/apache/incubator-pulsar/pull/1662) NPE when cursor failed to close empty subscription + * [#1370](https://github.com/apache/incubator-pulsar/pull/1370) Relocate service files for shading pulsar-client-admin module + * [#1265](https://github.com/apache/incubator-pulsar/pull/1265) Fixed lookup redirect logic on Proxyside + * [#1428](https://github.com/apache/incubator-pulsar/pull/1428) Handle Race condition in concurrent bundle split + * [#1817](https://github.com/apache/incubator-pulsar/pull/1817) Fixed mem leak when acknowledging while disconnected from broke + * [#1851](https://github.com/apache/incubator-pulsar/pull/1851) Fixing resource leak due to open file descriptors in SecurityUtility. + +The complete list of changes can be found at: +https://github.com/apache/incubator-pulsar/milestone/15?closed=1 + +https://github.com/apache/incubator-pulsar/releases/tag/v1.22.1-incubating + ### 2.0.0-rc1-incubating 2018-05-29 This is the fifth release of Apache Pulsar since entering the ASF incubator and the
[GitHub] hrsakai commented on a change in pull request #1977: Enable listener to receive messages even if receiver queue size is zero
hrsakai commented on a change in pull request #1977: Enable listener to receive
messages even if receiver queue size is zero
URL: https://github.com/apache/incubator-pulsar/pull/1977#discussion_r196281235
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
##
@@ -722,12 +726,12 @@ void messageReceived(MessageIdData messageId, ByteBuf
headersAndPayload, ClientC
// if the conf.getReceiverQueueSize() is 0 then discard
message if no one is waiting for it.
// if asyncReceive is waiting then notify callback without
adding to incomingMessages queue
unAckedMessageTracker.add((MessageIdImpl)
message.getMessageId());
-boolean asyncReceivedWaiting = !pendingReceives.isEmpty();
-if ((conf.getReceiverQueueSize() != 0 ||
waitingOnReceiveForZeroQueueSize) && !asyncReceivedWaiting) {
-incomingMessages.add(message);
-}
-if (asyncReceivedWaiting) {
+if (!pendingReceives.isEmpty()) {
notifyPendingReceivedCallback(message, null);
+} else if (conf.getReceiverQueueSize() != 0 ||
waitingOnReceiveForZeroQueueSize) {
+incomingMessages.add(message);
+} else if (conf.getReceiverQueueSize() == 0 && listener !=
null) {
+triggerZeroQueueSizeListener(message);
Review comment:
I see.
Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] massakam commented on a change in pull request #1977: Enable listener to receive messages even if receiver queue size is zero
massakam commented on a change in pull request #1977: Enable listener to
receive messages even if receiver queue size is zero
URL: https://github.com/apache/incubator-pulsar/pull/1977#discussion_r196277383
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
##
@@ -722,12 +726,12 @@ void messageReceived(MessageIdData messageId, ByteBuf
headersAndPayload, ClientC
// if the conf.getReceiverQueueSize() is 0 then discard
message if no one is waiting for it.
// if asyncReceive is waiting then notify callback without
adding to incomingMessages queue
unAckedMessageTracker.add((MessageIdImpl)
message.getMessageId());
-boolean asyncReceivedWaiting = !pendingReceives.isEmpty();
-if ((conf.getReceiverQueueSize() != 0 ||
waitingOnReceiveForZeroQueueSize) && !asyncReceivedWaiting) {
-incomingMessages.add(message);
-}
-if (asyncReceivedWaiting) {
+if (!pendingReceives.isEmpty()) {
notifyPendingReceivedCallback(message, null);
+} else if (conf.getReceiverQueueSize() != 0 ||
waitingOnReceiveForZeroQueueSize) {
+incomingMessages.add(message);
+} else if (conf.getReceiverQueueSize() == 0 && listener !=
null) {
+triggerZeroQueueSizeListener(message);
Review comment:
It is because that a flow command is sent before the listener finishes
processing of the message if existing logic is used. The flow command should be
sent after the listener completes processing of the message.
In addition, I think that it is not preferable to add the message to
`incommingMessages` if the queue size is zero.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] massakam commented on a change in pull request #1977: Enable listener to receive messages even if receiver queue size is zero
massakam commented on a change in pull request #1977: Enable listener to
receive messages even if receiver queue size is zero
URL: https://github.com/apache/incubator-pulsar/pull/1977#discussion_r196277383
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
##
@@ -722,12 +726,12 @@ void messageReceived(MessageIdData messageId, ByteBuf
headersAndPayload, ClientC
// if the conf.getReceiverQueueSize() is 0 then discard
message if no one is waiting for it.
// if asyncReceive is waiting then notify callback without
adding to incomingMessages queue
unAckedMessageTracker.add((MessageIdImpl)
message.getMessageId());
-boolean asyncReceivedWaiting = !pendingReceives.isEmpty();
-if ((conf.getReceiverQueueSize() != 0 ||
waitingOnReceiveForZeroQueueSize) && !asyncReceivedWaiting) {
-incomingMessages.add(message);
-}
-if (asyncReceivedWaiting) {
+if (!pendingReceives.isEmpty()) {
notifyPendingReceivedCallback(message, null);
+} else if (conf.getReceiverQueueSize() != 0 ||
waitingOnReceiveForZeroQueueSize) {
+incomingMessages.add(message);
+} else if (conf.getReceiverQueueSize() == 0 && listener !=
null) {
+triggerZeroQueueSizeListener(message);
Review comment:
It is because that a flow command is sent before the listener finishes
processing of the message if existing logic is used. The flow command should be
sent after the listener completes processing of the message.
In addition, I think that it is not preferable to add the message to
`incommingMessages` even though the queue size is zero.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[incubator-pulsar] branch master updated: Enable listener to receive messages even if receiver queue size is zero (#1977)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new c293ae3 Enable listener to receive messages even if receiver queue
size is zero (#1977)
c293ae3 is described below
commit c293ae33c28fad12c2ff3e90591e2a18ce95e97e
Author: massakam
AuthorDate: Tue Jun 19 10:54:53 2018 +0900
Enable listener to receive messages even if receiver queue size is zero
(#1977)
* Enable listener to receive messages even if receiver queue size is zero
* Send flow if acked message is received
---
.gitignore | 1 +
.../pulsar/client/impl/ZeroQueueSizeTest.java | 48 ++
.../apache/pulsar/client/impl/ConsumerImpl.java| 39 ++
3 files changed, 81 insertions(+), 7 deletions(-)
diff --git a/.gitignore b/.gitignore
index b009375..ed5a8b6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,6 +7,7 @@ logs
/data
pulsar-broker/tmp.*
pulsar-broker/src/test/resources/log4j2.yaml
+pulsar-functions/worker/test-tenant/
*.log
*.versionsBackup
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ZeroQueueSizeTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ZeroQueueSizeTest.java
index 725fa46..2859047 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ZeroQueueSizeTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ZeroQueueSizeTest.java
@@ -20,8 +20,12 @@ package org.apache.pulsar.client.impl;
import static org.testng.Assert.assertEquals;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
+import com.google.common.collect.Lists;
+
import org.apache.pulsar.broker.service.BrokerTestBase;
import org.apache.pulsar.client.admin.PulsarAdminException;
import org.apache.pulsar.client.api.Consumer;
@@ -123,6 +127,50 @@ public class ZeroQueueSizeTest extends BrokerTestBase {
}
@Test()
+public void zeroQueueSizeConsumerListener() throws Exception {
+String key = "zeroQueueSizeConsumerListener";
+
+// 1. Config
+final String topicName = "persistent://prop/use/ns-abc/topic-" + key;
+final String subscriptionName = "my-ex-subscription-" + key;
+final String messagePredicate = "my-message-" + key + "-";
+
+// 2. Create Producer
+Producer producer = pulsarClient.newProducer().topic(topicName)
+.enableBatching(false)
+.messageRoutingMode(MessageRoutingMode.SinglePartition)
+.create();
+
+// 3. Create Consumer
+List> messages = Lists.newArrayList();
+CountDownLatch latch = new CountDownLatch(totalMessages);
+ConsumerImpl consumer = (ConsumerImpl)
pulsarClient.newConsumer().topic(topicName)
+
.subscriptionName(subscriptionName).receiverQueueSize(0).messageListener((cons,
msg) -> {
+assertEquals(((ConsumerImpl) cons).numMessagesInQueue(),
0);
+synchronized(messages) {
+messages.add(msg);
+}
+log.info("Consumer received: " + new
String(msg.getData()));
+latch.countDown();
+}).subscribe();
+
+// 3. producer publish messages
+for (int i = 0; i < totalMessages; i++) {
+String message = messagePredicate + i;
+log.info("Producer produced: " + message);
+producer.send(message.getBytes());
+}
+
+// 4. Receiver receives the message
+latch.await();
+assertEquals(consumer.numMessagesInQueue(), 0);
+assertEquals(messages.size(), totalMessages);
+for (int i = 0; i < messages.size(); i++) {
+assertEquals(new String(messages.get(i).getData()),
messagePredicate + i);
+}
+}
+
+@Test()
public void zeroQueueSizeSharedSubscription() throws PulsarClientException
{
String key = "zeroQueueSizeSharedSubscription";
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
index 056b17e..10fe6ff 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
@@ -485,7 +485,8 @@ public class ConsumerImpl extends ConsumerBase
implements ConnectionHandle
AVAILABLE_PERMITS_UPDATER.set(this, 0);
// For zerosize queue : If the connection is reset and
someone is waiting for the messages
// or queue was not empty: send a flow command
-if (waitingOnReceiveForZeroQueueSize ||
[GitHub] merlimat closed pull request #1977: Enable listener to receive messages even if receiver queue size is zero
merlimat closed pull request #1977: Enable listener to receive messages even if
receiver queue size is zero
URL: https://github.com/apache/incubator-pulsar/pull/1977
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.gitignore b/.gitignore
index b009375637..ed5a8b63ea 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,6 +7,7 @@ logs
/data
pulsar-broker/tmp.*
pulsar-broker/src/test/resources/log4j2.yaml
+pulsar-functions/worker/test-tenant/
*.log
*.versionsBackup
diff --git
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ZeroQueueSizeTest.java
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ZeroQueueSizeTest.java
index 725fa46927..2859047fae 100644
---
a/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ZeroQueueSizeTest.java
+++
b/pulsar-broker/src/test/java/org/apache/pulsar/client/impl/ZeroQueueSizeTest.java
@@ -20,8 +20,12 @@
import static org.testng.Assert.assertEquals;
+import java.util.List;
+import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
+import com.google.common.collect.Lists;
+
import org.apache.pulsar.broker.service.BrokerTestBase;
import org.apache.pulsar.client.admin.PulsarAdminException;
import org.apache.pulsar.client.api.Consumer;
@@ -122,6 +126,50 @@ public void zeroQueueSizeNormalConsumer() throws
PulsarClientException {
}
}
+@Test()
+public void zeroQueueSizeConsumerListener() throws Exception {
+String key = "zeroQueueSizeConsumerListener";
+
+// 1. Config
+final String topicName = "persistent://prop/use/ns-abc/topic-" + key;
+final String subscriptionName = "my-ex-subscription-" + key;
+final String messagePredicate = "my-message-" + key + "-";
+
+// 2. Create Producer
+Producer producer = pulsarClient.newProducer().topic(topicName)
+.enableBatching(false)
+.messageRoutingMode(MessageRoutingMode.SinglePartition)
+.create();
+
+// 3. Create Consumer
+List> messages = Lists.newArrayList();
+CountDownLatch latch = new CountDownLatch(totalMessages);
+ConsumerImpl consumer = (ConsumerImpl)
pulsarClient.newConsumer().topic(topicName)
+
.subscriptionName(subscriptionName).receiverQueueSize(0).messageListener((cons,
msg) -> {
+assertEquals(((ConsumerImpl) cons).numMessagesInQueue(),
0);
+synchronized(messages) {
+messages.add(msg);
+}
+log.info("Consumer received: " + new
String(msg.getData()));
+latch.countDown();
+}).subscribe();
+
+// 3. producer publish messages
+for (int i = 0; i < totalMessages; i++) {
+String message = messagePredicate + i;
+log.info("Producer produced: " + message);
+producer.send(message.getBytes());
+}
+
+// 4. Receiver receives the message
+latch.await();
+assertEquals(consumer.numMessagesInQueue(), 0);
+assertEquals(messages.size(), totalMessages);
+for (int i = 0; i < messages.size(); i++) {
+assertEquals(new String(messages.get(i).getData()),
messagePredicate + i);
+}
+}
+
@Test()
public void zeroQueueSizeSharedSubscription() throws PulsarClientException
{
String key = "zeroQueueSizeSharedSubscription";
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
index 056b17e9f9..10fe6ff785 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
@@ -485,7 +485,8 @@ public void connectionOpened(final ClientCnx cnx) {
AVAILABLE_PERMITS_UPDATER.set(this, 0);
// For zerosize queue : If the connection is reset and
someone is waiting for the messages
// or queue was not empty: send a flow command
-if (waitingOnReceiveForZeroQueueSize ||
(conf.getReceiverQueueSize() == 0 && currentSize > 0)) {
+if (waitingOnReceiveForZeroQueueSize
+|| (conf.getReceiverQueueSize() == 0 &&
(currentSize > 0 || listener != null))) {
sendFlowPermitsToBroker(cnx, 1);
}
} else {
@@ -678,6 +679,9 @@ void messageReceived(MessageIdData messageId, ByteBuf
headersAndPayload, ClientC
log.debug("[{}][{}] Ignoring message as it was already being
acked earlier by same consumer {}/{}",
[GitHub] rdhabalia removed a comment on issue #1509: Fix init of Input CustomSerDe of function
rdhabalia removed a comment on issue #1509: Fix init of Input CustomSerDe of function URL: https://github.com/apache/incubator-pulsar/pull/1509#issuecomment-398246220 As a user, I felt it was a bug because we can pass This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1984: Remove `--advertised-address` from docker getting-started section
merlimat commented on issue #1984: Remove `--advertised-address` from docker getting-started section URL: https://github.com/apache/incubator-pulsar/pull/1984#issuecomment-398245578 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Add documentation for schema CLI (#1983)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 4ed4233 Add documentation for schema CLI (#1983)
4ed4233 is described below
commit 4ed4233d9ed0e119b679cb120a014cd456967ac9
Author: Sijie Guo
AuthorDate: Mon Jun 18 18:29:24 2018 -0700
Add documentation for schema CLI (#1983)
*Motivation*
Make sure we document schema CLI
*Solution*
Add the section in pulsar-admin for schemas CLI.
The last `compact` section is duplicated so removed it.
---
site/_data/cli/pulsar-admin.yaml | 23 ++-
1 file changed, 18 insertions(+), 5 deletions(-)
diff --git a/site/_data/cli/pulsar-admin.yaml b/site/_data/cli/pulsar-admin.yaml
index dbbbf79..46fc0a4 100644
--- a/site/_data/cli/pulsar-admin.yaml
+++ b/site/_data/cli/pulsar-admin.yaml
@@ -730,9 +730,22 @@ commands:
description: A bundle of the form `{start-boundary}_{end_boundary}`.
- flags: -n, --namespace
description: The namespace as `property/cluster/namespace`, for example
`my-prop/my-cluster/my-ns`.
-- name: topics
- description: Operations related to Pulsar topics of all kinds (both
persistent and non-persistent)
+- name: schemas
+ description: Operations related to Schemas associated with Pulsar topics.
subcommands:
- - name: compact
-description: Runs a compaction operation on the specified topic
-argument: topic-name
+ - name: upload
+description: Upload the schema definition for a topic
+argument: persistent://tenant/namespace/topic
+options:
+- flags: --filename
+ description: The path to the schema definition file. An example schema
file is available under `conf` directory.
+ - name: delete
+description: Delete the schema definition associated with a topic
+argument: persistent://tenant/namespace/topic
+ - name: get
+description: Retrieve the schema definition assoicated with a topic (at a
given version if version is supplied).
+argument: persistent://tenant/namespace/topic
+options:
+- flags: --version
+ description: The version of the schema definition to retrive for a topic.
+
[GitHub] merlimat closed pull request #1983: Add documentation for schema CLI
merlimat closed pull request #1983: Add documentation for schema CLI
URL: https://github.com/apache/incubator-pulsar/pull/1983
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/site/_data/cli/pulsar-admin.yaml b/site/_data/cli/pulsar-admin.yaml
index dbbbf794a1..46fc0a4fac 100644
--- a/site/_data/cli/pulsar-admin.yaml
+++ b/site/_data/cli/pulsar-admin.yaml
@@ -730,9 +730,22 @@ commands:
description: A bundle of the form `{start-boundary}_{end_boundary}`.
- flags: -n, --namespace
description: The namespace as `property/cluster/namespace`, for example
`my-prop/my-cluster/my-ns`.
-- name: topics
- description: Operations related to Pulsar topics of all kinds (both
persistent and non-persistent)
+- name: schemas
+ description: Operations related to Schemas associated with Pulsar topics.
subcommands:
- - name: compact
-description: Runs a compaction operation on the specified topic
-argument: topic-name
+ - name: upload
+description: Upload the schema definition for a topic
+argument: persistent://tenant/namespace/topic
+options:
+- flags: --filename
+ description: The path to the schema definition file. An example schema
file is available under `conf` directory.
+ - name: delete
+description: Delete the schema definition associated with a topic
+argument: persistent://tenant/namespace/topic
+ - name: get
+description: Retrieve the schema definition assoicated with a topic (at a
given version if version is supplied).
+argument: persistent://tenant/namespace/topic
+options:
+- flags: --version
+ description: The version of the schema definition to retrive for a topic.
+
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] merlimat closed pull request #1982: Issue 1918: Improve message id comparison
merlimat closed pull request #1982: Issue 1918: Improve message id comparison
URL: https://github.com/apache/incubator-pulsar/pull/1982
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageIdImpl.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageIdImpl.java
index d87a6ab91b..78109c490c 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageIdImpl.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageIdImpl.java
@@ -72,6 +72,8 @@ public int compareTo(MessageId o) {
} else {
return res;
}
+} else if (o instanceof TopicMessageIdImpl) {
+return compareTo(((TopicMessageIdImpl) o).getInnerMessageId());
} else {
throw new IllegalArgumentException(
"expected BatchMessageIdImpl object. Got instance of " +
o.getClass().getName());
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java
index 4810f0c921..41cc9c9d50 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java
@@ -73,6 +73,9 @@ public boolean equals(Object obj) {
if (obj instanceof MessageIdImpl) {
MessageIdImpl other = (MessageIdImpl) obj;
return ledgerId == other.ledgerId && entryId == other.entryId &&
partitionIndex == other.partitionIndex;
+} else if (obj instanceof BatchMessageIdImpl){
+BatchMessageIdImpl other = (BatchMessageIdImpl) obj;
+return other.equals(this);
}
return false;
}
@@ -148,16 +151,18 @@ public static MessageId fromByteArray(byte[] data) throws
IOException {
@Override
public int compareTo(MessageId o) {
-if (!(o instanceof MessageIdImpl)) {
+if (o instanceof MessageIdImpl) {
+MessageIdImpl other = (MessageIdImpl) o;
+return ComparisonChain.start()
+.compare(this.ledgerId, other.ledgerId)
+.compare(this.entryId, other.entryId)
+.compare(this.getPartitionIndex(), other.getPartitionIndex())
+.result();
+} else if (o instanceof TopicMessageIdImpl) {
+return compareTo(((TopicMessageIdImpl) o).getInnerMessageId());
+} else {
throw new IllegalArgumentException(
"expected MessageIdImpl object. Got instance of " +
o.getClass().getName());
}
-
-MessageIdImpl other = (MessageIdImpl) o;
-return ComparisonChain.start()
-.compare(this.ledgerId, other.ledgerId)
-.compare(this.entryId, other.entryId)
-.compare(this.getPartitionIndex(), other.getPartitionIndex())
-.result();
}
}
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicMessageIdImpl.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicMessageIdImpl.java
index c7cc4536ad..071b804ffb 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicMessageIdImpl.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicMessageIdImpl.java
@@ -18,6 +18,7 @@
*/
package org.apache.pulsar.client.impl;
+import java.util.Objects;
import org.apache.pulsar.client.api.MessageId;
public class TopicMessageIdImpl implements MessageId {
@@ -42,6 +43,16 @@ public MessageId getInnerMessageId() {
return messageId.toByteArray();
}
+@Override
+public boolean equals(Object obj) {
+if (!(obj instanceof TopicMessageIdImpl)) {
+return false;
+}
+TopicMessageIdImpl other = (TopicMessageIdImpl) obj;
+return Objects.equals(topicName, other.topicName)
+&& Objects.equals(messageId, other.messageId);
+}
+
@Override
public int compareTo(MessageId o) {
return messageId.compareTo(o);
diff --git
a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/MessageIdCompareToTest.java
b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/MessageIdCompareToTest.java
index 690fc897c0..78af44efba 100644
---
a/pulsar-client/src/test/java/org/apache/pulsar/client/impl/MessageIdCompareToTest.java
+++
b/pulsar-client/src/test/java/org/apache/pulsar/client/impl/MessageIdCompareToTest.java
@@ -18,6 +18,7 @@
*/
package org.apache.pulsar.client.impl;
+import static org.testng.Assert.assertFalse;
import static org.testng.Assert.assertTrue;
import org.testng.annotations.Test;
@@
[incubator-pulsar] branch master updated: Improve message id comparison (#1982)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 374d2ac Improve message id comparison (#1982)
374d2ac is described below
commit 374d2ac51c99a61c9895b1fec095bbb0439b722c
Author: Sijie Guo
AuthorDate: Mon Jun 18 18:29:10 2018 -0700
Improve message id comparison (#1982)
*Motivation*
Fixes #1918
*Solution*
Make sure `MessageIdImpl` and `BatchMessageIdImpl` recognize
TopicMessageIdImpl
---
.../pulsar/client/impl/BatchMessageIdImpl.java | 2 ++
.../apache/pulsar/client/impl/MessageIdImpl.java | 21 ++-
.../pulsar/client/impl/TopicMessageIdImpl.java | 11 ++
.../pulsar/client/impl/MessageIdCompareToTest.java | 42 ++
4 files changed, 68 insertions(+), 8 deletions(-)
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageIdImpl.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageIdImpl.java
index d87a6ab..78109c4 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageIdImpl.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/BatchMessageIdImpl.java
@@ -72,6 +72,8 @@ public class BatchMessageIdImpl extends MessageIdImpl {
} else {
return res;
}
+} else if (o instanceof TopicMessageIdImpl) {
+return compareTo(((TopicMessageIdImpl) o).getInnerMessageId());
} else {
throw new IllegalArgumentException(
"expected BatchMessageIdImpl object. Got instance of " +
o.getClass().getName());
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java
index 4810f0c..41cc9c9 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageIdImpl.java
@@ -73,6 +73,9 @@ public class MessageIdImpl implements MessageId {
if (obj instanceof MessageIdImpl) {
MessageIdImpl other = (MessageIdImpl) obj;
return ledgerId == other.ledgerId && entryId == other.entryId &&
partitionIndex == other.partitionIndex;
+} else if (obj instanceof BatchMessageIdImpl){
+BatchMessageIdImpl other = (BatchMessageIdImpl) obj;
+return other.equals(this);
}
return false;
}
@@ -148,16 +151,18 @@ public class MessageIdImpl implements MessageId {
@Override
public int compareTo(MessageId o) {
-if (!(o instanceof MessageIdImpl)) {
+if (o instanceof MessageIdImpl) {
+MessageIdImpl other = (MessageIdImpl) o;
+return ComparisonChain.start()
+.compare(this.ledgerId, other.ledgerId)
+.compare(this.entryId, other.entryId)
+.compare(this.getPartitionIndex(), other.getPartitionIndex())
+.result();
+} else if (o instanceof TopicMessageIdImpl) {
+return compareTo(((TopicMessageIdImpl) o).getInnerMessageId());
+} else {
throw new IllegalArgumentException(
"expected MessageIdImpl object. Got instance of " +
o.getClass().getName());
}
-
-MessageIdImpl other = (MessageIdImpl) o;
-return ComparisonChain.start()
-.compare(this.ledgerId, other.ledgerId)
-.compare(this.entryId, other.entryId)
-.compare(this.getPartitionIndex(), other.getPartitionIndex())
-.result();
}
}
diff --git
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicMessageIdImpl.java
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicMessageIdImpl.java
index c7cc453..071b804 100644
---
a/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicMessageIdImpl.java
+++
b/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TopicMessageIdImpl.java
@@ -18,6 +18,7 @@
*/
package org.apache.pulsar.client.impl;
+import java.util.Objects;
import org.apache.pulsar.client.api.MessageId;
public class TopicMessageIdImpl implements MessageId {
@@ -43,6 +44,16 @@ public class TopicMessageIdImpl implements MessageId {
}
@Override
+public boolean equals(Object obj) {
+if (!(obj instanceof TopicMessageIdImpl)) {
+return false;
+}
+TopicMessageIdImpl other = (TopicMessageIdImpl) obj;
+return Objects.equals(topicName, other.topicName)
+&& Objects.equals(messageId, other.messageId);
+}
+
+@Override
public int compareTo(MessageId o) {
return messageId.compareTo(o);
}
diff --git
[GitHub] merlimat closed issue #1918: Improve equals method of MessageId
merlimat closed issue #1918: Improve equals method of MessageId URL: https://github.com/apache/incubator-pulsar/issues/1918 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on a change in pull request #1985: Added release notes for 2.0.1-incubating release
merlimat commented on a change in pull request #1985: Added release notes for 2.0.1-incubating release URL: https://github.com/apache/incubator-pulsar/pull/1985#discussion_r196269824 ## File path: site/release-notes.md ## @@ -26,6 +26,42 @@ layout: content ## Apache incubator +### 2.0.1-incubating 2018-06-18 + +This release fixes issues reported for 2.0.0-rc1-incubating. + + * [#1893](https://github.com/apache/incubator-pulsar/pull/1893) - Fixed issues with Python packages on PyPI + * [#1797](https://github.com/apache/incubator-pulsar/issues/1797) - Proxy doesn't strip the request +URL for admin requests correctly + * [#1862](https://github.com/apache/incubator-pulsar/pull/1862) - Fix REST APIs provided by Pulsar proxy + +The complete list of changes can be found at: +https://github.com/apache/incubator-pulsar/milestone/14?closed=1 + +https://github.com/apache/incubator-pulsar/releases/tag/v2.0.0-rc1-incubating Review comment: Nice catch! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] hrsakai commented on a change in pull request #1977: Enable listener to receive messages even if receiver queue size is zero
hrsakai commented on a change in pull request #1977: Enable listener to receive
messages even if receiver queue size is zero
URL: https://github.com/apache/incubator-pulsar/pull/1977#discussion_r196266417
##
File path:
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
##
@@ -722,12 +726,12 @@ void messageReceived(MessageIdData messageId, ByteBuf
headersAndPayload, ClientC
// if the conf.getReceiverQueueSize() is 0 then discard
message if no one is waiting for it.
// if asyncReceive is waiting then notify callback without
adding to incomingMessages queue
unAckedMessageTracker.add((MessageIdImpl)
message.getMessageId());
-boolean asyncReceivedWaiting = !pendingReceives.isEmpty();
-if ((conf.getReceiverQueueSize() != 0 ||
waitingOnReceiveForZeroQueueSize) && !asyncReceivedWaiting) {
-incomingMessages.add(message);
-}
-if (asyncReceivedWaiting) {
+if (!pendingReceives.isEmpty()) {
notifyPendingReceivedCallback(message, null);
+} else if (conf.getReceiverQueueSize() != 0 ||
waitingOnReceiveForZeroQueueSize) {
+incomingMessages.add(message);
+} else if (conf.getReceiverQueueSize() == 0 && listener !=
null) {
+triggerZeroQueueSizeListener(message);
Review comment:
why do you call `triggerZeroQueueSizeListener()` instead of adding message
to incommingMessages?
you can't use [existing
logic](https://github.com/massakam/pulsar/blob/c1ab9d0edfec283c46e346d6d48b6134542bd01f/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L761-L792
)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] nkurihar commented on a change in pull request #1985: Added release notes for 2.0.1-incubating release
nkurihar commented on a change in pull request #1985: Added release notes for 2.0.1-incubating release URL: https://github.com/apache/incubator-pulsar/pull/1985#discussion_r196265057 ## File path: site/release-notes.md ## @@ -26,6 +26,42 @@ layout: content ## Apache incubator +### 2.0.1-incubating 2018-06-18 + +This release fixes issues reported for 2.0.0-rc1-incubating. + + * [#1893](https://github.com/apache/incubator-pulsar/pull/1893) - Fixed issues with Python packages on PyPI + * [#1797](https://github.com/apache/incubator-pulsar/issues/1797) - Proxy doesn't strip the request +URL for admin requests correctly + * [#1862](https://github.com/apache/incubator-pulsar/pull/1862) - Fix REST APIs provided by Pulsar proxy + +The complete list of changes can be found at: +https://github.com/apache/incubator-pulsar/milestone/14?closed=1 + +https://github.com/apache/incubator-pulsar/releases/tag/v2.0.0-rc1-incubating Review comment: URL seems wrong: https://github.com/apache/incubator-pulsar/releases/tag/v2.0.1-incubating This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] 01/01: Fixed python extension build
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch fix-python-build
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
commit f0bd71a57aaf1e7738f07dd732c22490ba15b210
Author: Matteo Merli
AuthorDate: Mon Jun 18 17:03:22 2018 -0700
Fixed python extension build
---
pulsar-client-cpp/docker/build-wheel-file-within-docker.sh | 14 +-
pulsar-client-cpp/docker/build-wheels.sh | 2 +-
pulsar-client-cpp/python/setup.py | 2 +-
3 files changed, 11 insertions(+), 7 deletions(-)
diff --git a/pulsar-client-cpp/docker/build-wheel-file-within-docker.sh
b/pulsar-client-cpp/docker/build-wheel-file-within-docker.sh
index d550ca9..d4cb9af 100755
--- a/pulsar-client-cpp/docker/build-wheel-file-within-docker.sh
+++ b/pulsar-client-cpp/docker/build-wheel-file-within-docker.sh
@@ -1,4 +1,5 @@
#!/bin/bash
+
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
@@ -18,19 +19,22 @@
# under the License.
#
-set -e
+set -e -x
cd /pulsar/pulsar-client-cpp
-find . -name CMakeCache.txt | xargs rm
-find . -name CMakeFiles | xargs rm -rf
+find . -name CMakeCache.txt | xargs -r rm
+find . -name CMakeFiles | xargs -r rm -rf
cmake .
-DPYTHON_INCLUDE_DIR=/opt/python/$PYTHON_SPEC/include/python$PYTHON_VERSION \
-DPYTHON_LIBRARY=/opt/python/$PYTHON_SPEC/lib \
--DLINK_STATIC=ON
+-DLINK_STATIC=ON \
+-DBUILD_TESTS=OFF
make clean
-make _pulsar
+make _pulsar -j3
cd python
python setup.py bdist_wheel
+
+auditwheel repair dist/pulsar_client-*-$PYTHON_SPEC-linux_x86_64.whl
diff --git a/pulsar-client-cpp/docker/build-wheels.sh
b/pulsar-client-cpp/docker/build-wheels.sh
index 35cb789..2146869 100755
--- a/pulsar-client-cpp/docker/build-wheels.sh
+++ b/pulsar-client-cpp/docker/build-wheels.sh
@@ -44,5 +44,5 @@ for line in "${PYTHON_VERSIONS[@]}"; do
IMAGE_NAME=$BUILD_IMAGE_NAME:manylinux-$PYTHON_SPEC
echo "Using image: $IMAGE_NAME"
-docker run -i -v $PWD:/pulsar $IMAGE_NAME
/build-wheel-file-within-docker.sh
+docker run -i -v $PWD:/pulsar $IMAGE_NAME
/pulsar/pulsar-client-cpp/docker/build-wheel-file-within-docker.sh
done
diff --git a/pulsar-client-cpp/python/setup.py
b/pulsar-client-cpp/python/setup.py
index ab91f0f..b20f3d6 100644
--- a/pulsar-client-cpp/python/setup.py
+++ b/pulsar-client-cpp/python/setup.py
@@ -60,7 +60,7 @@ class my_build_ext(build_ext.build_ext):
setup(
name="pulsar-client",
version=VERSION,
-py_modules=['pulsar'],
+packages=['pulsar', 'pulsar.functions'],
cmdclass={'build_ext': my_build_ext},
ext_modules=[Extension('_pulsar', [])],
[incubator-pulsar] branch fix-python-build updated (ca83177 -> f0bd71a)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a change to branch fix-python-build in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git. discard ca83177 Fixed python extension build new f0bd71a Fixed python extension build This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (ca83177) \ N -- N -- N refs/heads/fix-python-build (f0bd71a) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: pulsar-client-cpp/docker/build-wheel-file-within-docker.sh | 2 ++ 1 file changed, 2 insertions(+)
[GitHub] merlimat commented on issue #1977: Enable listener to receive messages even if receiver queue size is zero
merlimat commented on issue #1977: Enable listener to receive messages even if receiver queue size is zero URL: https://github.com/apache/incubator-pulsar/pull/1977#issuecomment-398223251 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat opened a new pull request #1985: Added release notes for 2.0.1-incubating release
merlimat opened a new pull request #1985: Added release notes for 2.0.1-incubating release URL: https://github.com/apache/incubator-pulsar/pull/1985 ### Motivation Updating website to with new release This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1737: Bookies at 100% CPU
sijie commented on issue #1737: Bookies at 100% CPU URL: https://github.com/apache/incubator-pulsar/issues/1737#issuecomment-398214676 @sschepens @merlimat any updates or actions here? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie opened a new pull request #1984: Remove `--advertised-address` from docker getting-started section
sijie opened a new pull request #1984: Remove `--advertised-address` from docker getting-started section URL: https://github.com/apache/incubator-pulsar/pull/1984 *Motivation* Fixes #1929 *Changes* Per [comment](https://github.com/apache/incubator-pulsar/issues/1929#issuecomment-395497432), we don't need to specify `--advertised-address` after 1.22.0 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1943: Non serializable client builder
sijie commented on issue #1943: Non serializable client builder URL: https://github.com/apache/incubator-pulsar/issues/1943#issuecomment-398212974 I think the original clients and client configuration are not removed. so people should be able to still use the old api to construct the client, where client configuration is still serializable, right? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie opened a new pull request #1983: Add documentation for schema CLI
sijie opened a new pull request #1983: Add documentation for schema CLI URL: https://github.com/apache/incubator-pulsar/pull/1983 *Motivation* Make sure we document schema CLI *Solution* Add the section in pulsar-admin for schemas CLI. The last `compact` section is duplicated so removed it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie opened a new pull request #1982: Issue 1918: Improve message id comparison
sijie opened a new pull request #1982: Issue 1918: Improve message id comparison URL: https://github.com/apache/incubator-pulsar/pull/1982 *Motivation* Fixes #1918 *Solution* Make sure `MessageIdImpl` and `BatchMessageIdImpl` recognize TopicMessageIdImpl This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1821: Closing client when Proxyconnection is disconnected
sijie commented on issue #1821: Closing client when Proxyconnection is disconnected URL: https://github.com/apache/incubator-pulsar/pull/1821#issuecomment-398197893 @jai1 great! Wednesday is good. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1981: Poor autorecovery performance on dedicated hosts
sijie commented on issue #1981: Poor autorecovery performance on dedicated hosts URL: https://github.com/apache/incubator-pulsar/issues/1981#issuecomment-398196526 @bardock so I think it is throttled by this setting: https://github.com/yahoo/bookkeeper/blob/yahoo-4.3/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/AbstractConfiguration.java#L189 `rereplicationEntryBatchSize` that says if you have 15 instances running autorecovery, for each instance it will only replicate one ledger at a time, for replicating a ledger, it will only replicate `rereplicationEntryBatchSize` at a time. I would suggest you increase this `rereplicationEntryBatchSize` to a larger value to speed up the recovery process. However you might consider doing some calculation to not use all the network bandwidth for rereplication. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
svn commit: r27537 - /dev/incubator/pulsar/pulsar-2.0.1-incubating-candidate-2/ /release/incubator/pulsar/pulsar-2.0.1-incubating/
Author: mmerli Date: Mon Jun 18 20:41:58 2018 New Revision: 27537 Log: Release pulsar-2.0.1-incubating Added: release/incubator/pulsar/pulsar-2.0.1-incubating/ - copied from r27536, dev/incubator/pulsar/pulsar-2.0.1-incubating-candidate-2/ Removed: dev/incubator/pulsar/pulsar-2.0.1-incubating-candidate-2/
[GitHub] jai1 commented on issue #1821: Closing client when Proxyconnection is disconnected
jai1 commented on issue #1821: Closing client when Proxyconnection is disconnected URL: https://github.com/apache/incubator-pulsar/pull/1821#issuecomment-398188483 If this is blocking the release then we can defer it to the next one, else I will fix the tests and rebase the PR by Wednesday This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] bardock opened a new issue #1981: Poor autorecovery performance on dedicated hosts
bardock opened a new issue #1981: Poor autorecovery performance on dedicated hosts URL: https://github.com/apache/incubator-pulsar/issues/1981 We have a cluster of 20 bookies and another cluster of 15 instances running autorecovery. We've just lost a bookie (physically) and right now there are 10k ledgers marked as underreplicated (approximately 100 GB of data). Autorecovery is replicating 50-100 ledgers per minute. Is there any way to improve this performance? There are enough resources (CPU, RAM and network bandwidth), however publish times increased from 10 to 100 ms (pct99). We are using EC2 instances ([i3.2xlarge](https://aws.amazon.com/ec2/instance-types/i3/#Product_Details) for bookies and [c5.large](https://aws.amazon.com/ec2/instance-types/c5/#Product_Details) for autorecovery) and bookkeeper version 4.3.1.91-yahoo with the default config except these settings: ``` dbStorage_readAheadCacheMaxSizeMb=4096, dbStorage_rocksDB_blockCacheSize=4294967296 flushInterval=1 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jai1 closed pull request #254: Java Client - Support for getting consumer stats from broker
jai1 closed pull request #254: Java Client - Support for getting consumer stats
from broker
URL: https://github.com/apache/incubator-pulsar/pull/254
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/pulsar-broker/src/test/java/com/yahoo/pulsar/client/api/BrokerConsumerStatsTest.java
b/pulsar-broker/src/test/java/com/yahoo/pulsar/client/api/BrokerConsumerStatsTest.java
new file mode 100644
index 00..d5fb66c236
--- /dev/null
+++
b/pulsar-broker/src/test/java/com/yahoo/pulsar/client/api/BrokerConsumerStatsTest.java
@@ -0,0 +1,219 @@
+/**
+ * Copyright 2016 Yahoo Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.yahoo.pulsar.client.api;
+
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import com.yahoo.pulsar.broker.service.BrokerTestBase;
+import com.yahoo.pulsar.client.admin.PulsarAdminException;
+import com.yahoo.pulsar.client.api.Consumer;
+import com.yahoo.pulsar.client.api.ConsumerConfiguration;
+import com.yahoo.pulsar.client.api.Message;
+import com.yahoo.pulsar.client.api.Producer;
+import com.yahoo.pulsar.client.api.ProducerConfiguration;
+import com.yahoo.pulsar.client.api.ProducerConfiguration.MessageRoutingMode;
+import com.yahoo.pulsar.client.api.PulsarClientException;
+import com.yahoo.pulsar.client.api.SubscriptionType;
+import com.yahoo.pulsar.client.impl.PartitionedBrokerConsumerStatsImpl;
+
+public class BrokerConsumerStatsTest extends BrokerTestBase {
+private static final Logger log =
LoggerFactory.getLogger(BrokerConsumerStatsTest.class);
+private String topicName = "persistent://prop/cluster/ns/topic-";
+
+@BeforeClass
+@Override
+public void setup() throws Exception {
+baseSetup();
+}
+
+@AfterClass
+@Override
+protected void cleanup() throws Exception {
+internalCleanup();
+}
+
+@Test()
+public void testSharedSubscriptionMessageBacklog() throws
PulsarClientException {
+int totalMessages = 50;
+String topicNamePostFix = "testSharedSubscriptionMessageBacklog";
+
+// 1. Create consumers
+ConsumerConfiguration conf = new ConsumerConfiguration();
+conf.setSubscriptionType(SubscriptionType.Shared);
+Consumer consumer1 = pulsarClient.subscribe(topicName +
topicNamePostFix, "my-subscriber-name", conf);
+Consumer consumer2 = pulsarClient.subscribe(topicName +
topicNamePostFix, "my-subscriber-name", conf);
+Consumer consumer3 = pulsarClient.subscribe(topicName +
topicNamePostFix, "my-subscriber-name", conf);
+
+// 2. Create producers and produce messages
+Producer producer = pulsarClient.createProducer(topicName +
topicNamePostFix, new ProducerConfiguration());
+for (int i = 0; i < totalMessages; i++) {
+String message = "my-message-" + i;
+producer.send(message.getBytes());
+}
+
+int count = 0;
+try {
+// 3. Consumer all messages routed to consumer1
+Message msg = null;
+do {
+msg = consumer1.receive(1, TimeUnit.SECONDS);
+if (msg != null) {
+consumer1.acknowledge(msg);
+count++;
+}
+} while(msg != null);
+
+// 4. Check subscription type
+
log.debug(consumer1.getBrokerConsumerStatsAsync().get().toString());
+
Assert.assertEquals(consumer1.getBrokerConsumerStatsAsync().get().getSubscriptionType(),
+SubscriptionType.Shared);
+
Assert.assertEquals(consumer1.getBrokerConsumerStatsAsync().get().getMsgBacklog(),
totalMessages - count);
+
+// 5. Consumer all messages routed to consumer2
+do {
+msg = consumer2.receive(1, TimeUnit.SECONDS);
+if (msg != null) {
+consumer2.acknowledge(msg);
+count++;
+}
+} while(msg != null);
+
+// 6. Check consumer2 backlog
+
[GitHub] jai1 closed pull request #1248: Start Proxy in TLS only mode.
jai1 closed pull request #1248: Start Proxy in TLS only mode.
URL: https://github.com/apache/incubator-pulsar/pull/1248
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/conf/broker.conf b/conf/broker.conf
index 59a19d7b8b..4eaddefb3f 100644
--- a/conf/broker.conf
+++ b/conf/broker.conf
@@ -29,13 +29,13 @@ globalZookeeperServers=
brokerServicePort=6650
# Broker data port for TLS
-brokerServicePortTls=6651
+brokerServicePortTls=
# Port to use to server HTTP request
webServicePort=8080
# Port to use to server HTTPS request
-webServicePortTls=8443
+webServicePortTls=
# Hostname or IP address the service binds on, default is 0.0.0.0.
bindAddress=0.0.0.0
@@ -192,9 +192,6 @@ proxyRoles=
# else it just accepts the originalPrincipal and authorizes it (if required).
authenticateOriginalAuthData=false
-# Enable TLS
-tlsEnabled=false
-
# Path for the TLS certificate file
tlsCertificateFilePath=
diff --git a/conf/discovery.conf b/conf/discovery.conf
index 49f499a080..4957d9429d 100644
--- a/conf/discovery.conf
+++ b/conf/discovery.conf
@@ -30,13 +30,13 @@ zookeeperSessionTimeoutMs=3
servicePort=6650
# Port to use to server binary-proto-tls request
-servicePortTls=6651
+servicePortTls=
# Port that discovery service listen on
webServicePort=8080
# Port to use to server HTTPS request
-webServicePortTls=8443
+webServicePortTls=
# Control whether to bind directly on localhost rather than on normal hostname
bindOnLocalhost=false
@@ -65,8 +65,6 @@ superUserRoles=
authorizationAllowWildcardsMatching=false
# --- TLS --- #
-# Enable TLS
-tlsEnabled=false
# Path for the TLS certificate file
tlsCertificateFilePath=
diff --git a/conf/proxy.conf b/conf/proxy.conf
index 384cca06b6..aa8626be8d 100644
--- a/conf/proxy.conf
+++ b/conf/proxy.conf
@@ -30,13 +30,13 @@ zookeeperSessionTimeoutMs=3
servicePort=6650
# Port to use to server binary-proto-tls request
-servicePortTls=6651
+servicePortTls=
# Port that discovery service listen on
webServicePort=8080
# Port to use to server HTTPS request
-webServicePortTls=8443
+webServicePortTls=
# Path for the file used to determine the rotation status for the
proxy-instance when responding
# to service discovery health checks
@@ -71,9 +71,6 @@ forwardAuthorizationCredentials=false
# --- TLS --- #
-# Enable TLS in the proxy
-tlsEnabledInProxy=false
-
# Enable TLS when talking with the brokers
tlsEnabledWithBroker=false
diff --git a/conf/websocket.conf b/conf/websocket.conf
index 0ceda6273d..e720a5678a 100644
--- a/conf/websocket.conf
+++ b/conf/websocket.conf
@@ -34,7 +34,7 @@ brokerServiceUrlTls=
# Port to use to server HTTP request
webServicePort=8080
# Port to use to server HTTPS request
-webServicePortTls=8443
+webServicePortTls=
# Path for the file used to determine the rotation status for the
proxy-instance when responding
# to service discovery health checks
@@ -85,9 +85,6 @@ anonymousUserRole=
### --- TLS --- ###
-# Enable TLS
-tlsEnabled=false
-
# Accept untrusted TLS certificate from client
tlsAllowInsecureConnection=false
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index f851f7dd93..aa0972383a 100644
---
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -43,12 +43,12 @@
// Global Zookeeper quorum connection string
@FieldContext(required = false)
private String globalZookeeperServers;
-private int brokerServicePort = 6650;
-private int brokerServicePortTls = 6651;
+private Integer brokerServicePort = 6650;
+private Integer brokerServicePortTls;
// Port to use to server HTTP request
-private int webServicePort = 8080;
+private Integer webServicePort = 8080;
// Port to use to server HTTPS request
-private int webServicePortTls = 8443;
+private Integer webServicePortTls;
// Hostname or IP address the service binds on.
private String bindAddress = "0.0.0.0";
@@ -186,8 +186,6 @@
private int maxConsumersPerSubscription = 0;
/* --- TLS --- /
-// Enable TLS
-private boolean tlsEnabled = false;
// Path for the TLS certificate file
private String tlsCertificateFilePath;
// Path for the TLS private key file
@@ -457,32 +455,32 @@ public void setGlobalZookeeperServers(String
globalZookeeperServers) {
this.globalZookeeperServers = globalZookeeperServers;
}
-public int getBrokerServicePort() {
-return brokerServicePort;
+public Optional
[GitHub] jai1 closed pull request #322: More than one connection per broker in C++ client
jai1 closed pull request #322: More than one connection per broker in C++ client
URL: https://github.com/apache/incubator-pulsar/pull/322
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/pulsar-client-cpp/include/pulsar/Client.h
b/pulsar-client-cpp/include/pulsar/Client.h
index 70123f609a..9178ff1b02 100644
--- a/pulsar-client-cpp/include/pulsar/Client.h
+++ b/pulsar-client-cpp/include/pulsar/Client.h
@@ -163,6 +163,18 @@ class ClientConfiguration {
ClientConfiguration& setTlsAllowInsecureConnection(bool allowInsecure);
bool isTlsAllowInsecureConnection() const;
+/*
+ * Set max number of of connections per broker.
+ * @param connectionsPerBroker - number of connections per brokers
+ * @note - if connections per broker < 1 then the value will be set as to
1.
+ */
+ClientConfiguration& setConnectionsPerBroker(size_t connectionsPerBroker);
+
+/*
+ * Get number of connections per broker.
+ */
+size_t getConnectionsPerBroker() const;
+
private:
const AuthenticationPtr& getAuthenticationPtr() const;
diff --git a/pulsar-client-cpp/lib/Client.cc b/pulsar-client-cpp/lib/Client.cc
index 27033a39b3..0a00f20c1b 100644
--- a/pulsar-client-cpp/lib/Client.cc
+++ b/pulsar-client-cpp/lib/Client.cc
@@ -41,6 +41,7 @@ struct ClientConfiguration::Impl {
bool useTls;
std::string tlsTrustCertsFilePath;
bool tlsAllowInsecureConnection;
+size_t connectionsPerBroker;
Impl() : authenticationPtr(AuthFactory::Disabled()),
ioThreads(1),
operationTimeoutSeconds(30),
@@ -48,6 +49,7 @@ struct ClientConfiguration::Impl {
concurrentLookupRequest(5000),
logConfFilePath(),
useTls(false),
+ connectionsPerBroker(1),
tlsAllowInsecureConnection(true) {}
};
@@ -163,6 +165,18 @@ const std::string&
ClientConfiguration::getLogConfFilePath() const {
return impl_->logConfFilePath;
}
+ClientConfiguration& ClientConfiguration::setConnectionsPerBroker(size_t
connectionsPerBroker) {
+if (connectionsPerBroker < 1) {
+LOG_ERROR("connectionsPerBroker set to 1");
+}
+impl_->connectionsPerBroker = std::max(1uL, connectionsPerBroker);
+return *this;
+}
+
+size_t ClientConfiguration::getConnectionsPerBroker() const {
+return impl_->connectionsPerBroker;
+}
+
/
Client::Client(const std::string& serviceUrl)
diff --git a/pulsar-client-cpp/lib/ClientImpl.cc
b/pulsar-client-cpp/lib/ClientImpl.cc
index 85086509c5..866ace7ff9 100644
--- a/pulsar-client-cpp/lib/ClientImpl.cc
+++ b/pulsar-client-cpp/lib/ClientImpl.cc
@@ -63,7 +63,7 @@ namespace pulsar {
ioExecutorProvider_(boost::make_shared(clientConfiguration.getIOThreads())),
listenerExecutorProvider_(boost::make_shared(clientConfiguration.getMessageListenerThreads())),
partitionListenerExecutorProvider_(boost::make_shared(clientConfiguration.getMessageListenerThreads())),
- pool_(clientConfiguration, ioExecutorProvider_,
clientConfiguration.getAuthenticationPtr(), poolConnections),
+ pool_(clientConfiguration, ioExecutorProvider_,
clientConfiguration.getAuthenticationPtr(), poolConnections,
clientConfiguration.getConnectionsPerBroker()),
producerIdGenerator_(0),
consumerIdGenerator_(0),
requestIdGenerator_(0) {
diff --git a/pulsar-client-cpp/lib/ConnectionPool.cc
b/pulsar-client-cpp/lib/ConnectionPool.cc
index 1fd07d0fb5..2d42668ebc 100644
--- a/pulsar-client-cpp/lib/ConnectionPool.cc
+++ b/pulsar-client-cpp/lib/ConnectionPool.cc
@@ -15,7 +15,6 @@
*/
#include "ConnectionPool.h"
-
#include "LogUtils.h"
DECLARE_LOG_OBJECT()
@@ -24,34 +23,45 @@ namespace pulsar {
ConnectionPool::ConnectionPool(const ClientConfiguration& conf,
ExecutorServiceProviderPtr executorProvider,
- const AuthenticationPtr& authentication, bool
poolConnections)
+ const AuthenticationPtr& authentication,
+ bool poolConnections, size_t
connectionsPerBroker)
: clientConfiguration_(conf),
executorProvider_(executorProvider),
authentication_(authentication),
pool_(),
poolConnections_(poolConnections),
- mutex_() {
+ mutex_(),
+ connectionsPerBroker(connectionsPerBroker) {
}
Future ConnectionPool::getConnectionAsync(
const std::string& endpoint) {
boost::unique_lock lock(mutex_);
-
+PoolMap::iterator cnxIt = pool_.end();
if (poolConnections_) {
-PoolMap::iterator cnxIt = pool_.find(endpoint);
+
[GitHub] srkukarni commented on issue #1845: Functions schema integration
srkukarni commented on issue #1845: Functions schema integration URL: https://github.com/apache/incubator-pulsar/pull/1845#issuecomment-398185035 lgtm! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #1371: Switch default digest type for BookKeeper to CRC32c
merlimat commented on issue #1371: Switch default digest type for BookKeeper to CRC32c URL: https://github.com/apache/incubator-pulsar/issues/1371#issuecomment-398184761 Yes This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1916: Intermittent failure in integration test TestS3Offload.testPublishOffloadAndConsumeViaCLI
sijie commented on issue #1916: Intermittent failure in integration test TestS3Offload.testPublishOffloadAndConsumeViaCLI URL: https://github.com/apache/incubator-pulsar/issues/1916#issuecomment-398184485 moved this to 2.2.0 since #1968 is in for capturing the zookeeper journals. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1371: Switch default digest type for BookKeeper to CRC32c
sijie commented on issue #1371: Switch default digest type for BookKeeper to CRC32c URL: https://github.com/apache/incubator-pulsar/issues/1371#issuecomment-398184167 @merlimat shall we do this in 2.1? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1300: Add read compacted flag in C++ / Python consumer configuration.
sijie commented on issue #1300: Add read compacted flag in C++ / Python consumer configuration. URL: https://github.com/apache/incubator-pulsar/issues/1300#issuecomment-398182491 This is implemented at #1858 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie closed issue #1300: Add read compacted flag in C++ / Python consumer configuration.
sijie closed issue #1300: Add read compacted flag in C++ / Python consumer configuration. URL: https://github.com/apache/incubator-pulsar/issues/1300 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1594: Introduce pulsar-replicator and kinesis-provider with pulsar-function
sijie commented on issue #1594: Introduce pulsar-replicator and kinesis-provider with pulsar-function URL: https://github.com/apache/incubator-pulsar/pull/1594#issuecomment-398182196 @rdhabalia if I understand, we don't need this PR anymore, right? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1509: Fix init of Input CustomSerDe of function
sijie commented on issue #1509: Fix init of Input CustomSerDe of function URL: https://github.com/apache/incubator-pulsar/pull/1509#issuecomment-398181103 @rdhabalia is this still needed? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1599: Throw illegal state exception instead of NPE in zk-cache when no session available
sijie commented on issue #1599: Throw illegal state exception instead of NPE in zk-cache when no session available URL: https://github.com/apache/incubator-pulsar/pull/1599#issuecomment-398180915 @merlimat any updates on this one? is it required for 2.1? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1749: [WIP] Pulsar IO documentation
sijie commented on issue #1749: [WIP] Pulsar IO documentation URL: https://github.com/apache/incubator-pulsar/pull/1749#issuecomment-398180600 @jerrypeng do you mind picking this PR? I don't think we have any documentation for pulsar I/O, do we? /cc @srkukarni This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1821: Closing client when Proxyconnection is disconnected
sijie commented on issue #1821: Closing client when Proxyconnection is disconnected URL: https://github.com/apache/incubator-pulsar/pull/1821#issuecomment-398180233 @jai1 @merlimat what is the conclusion here? are we going to include this for 2.1? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1861: Cpp client: add getLastMessageId and hasMessageAvailable in consmer and reader
sijie commented on issue #1861: Cpp client: add getLastMessageId and hasMessageAvailable in consmer and reader URL: https://github.com/apache/incubator-pulsar/pull/1861#issuecomment-398179691 ping @zhaijack ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1838: Pulsar Go client docs
sijie commented on issue #1838: Pulsar Go client docs URL: https://github.com/apache/incubator-pulsar/pull/1838#issuecomment-398179529 @merlimat @hrsakai is this PR ready to go? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1956: Add function-package-url support in function cli add url support to function cli
sijie commented on issue #1956: Add function-package-url support in function cli add url support to function cli URL: https://github.com/apache/incubator-pulsar/pull/1956#issuecomment-398179233 @jerrypeng @srkukarni can you review the latest change? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie commented on issue #1845: Functions schema integration
sijie commented on issue #1845: Functions schema integration URL: https://github.com/apache/incubator-pulsar/pull/1845#issuecomment-398179063 @jerrypeng @srkukarni can you guys review this PR again? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Add documentation and flexible aws-credential plugin to support aws-role (#1972)
This is an automated email from the ASF dual-hosted git repository.
rdhabalia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 8ba8acd Add documentation and flexible aws-credential plugin to
support aws-role (#1972)
8ba8acd is described below
commit 8ba8acdbe812725df6739c5d19db3e47a553f351
Author: Rajan Dhabalia
AuthorDate: Mon Jun 18 12:15:28 2018 -0700
Add documentation and flexible aws-credential plugin to support aws-role
(#1972)
* Add documentation and flexible aws-credential plugin to support aws-role
* add protobuf relocation
---
pulsar-io/kinesis/pom.xml | 11 +-
.../io/kinesis/AwsCredentialProviderPlugin.java| 36 ++---
.../org/apache/pulsar/io/kinesis/KinesisSink.java | 104 --
.../apache/pulsar/io/kinesis/KinesisSinkTest.java | 149 +
4 files changed, 227 insertions(+), 73 deletions(-)
diff --git a/pulsar-io/kinesis/pom.xml b/pulsar-io/kinesis/pom.xml
index 7f9f488..0dee750 100644
--- a/pulsar-io/kinesis/pom.xml
+++ b/pulsar-io/kinesis/pom.xml
@@ -47,13 +47,16 @@
com.fasterxml.jackson.core
jackson-databind
- ${jackson.version}
com.fasterxml.jackson.dataformat
jackson-dataformat-yaml
- ${jackson.version}
+
+
+
+ com.google.code.gson
+ gson
@@ -110,10 +113,6 @@
com.google.protobuf
org.apache.pulsar.replicator.com.google.protobuf
-
- com.amazonaws
-
org.apache.pulsar.com.amazonaws
-
diff --git
a/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/AwsCredentialProviderPlugin.java
b/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/AwsCredentialProviderPlugin.java
index 8c616cc..7e463bb 100644
---
a/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/AwsCredentialProviderPlugin.java
+++
b/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/AwsCredentialProviderPlugin.java
@@ -19,42 +19,32 @@
package org.apache.pulsar.io.kinesis;
+import java.io.Closeable;
+
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.BasicSessionCredentials;
+
/**
* Kinesis source/sink calls credential-provider while refreshing aws
accessKey and secreKey. So, implementation
* AwsCredentialProviderPlugin needs to makes sure to return non-expired keys
when it requires.
*
*/
-public interface AwsCredentialProviderPlugin {
-
+public interface AwsCredentialProviderPlugin extends Closeable {
+
/**
* accepts aws-account related param and initialize credential provider.
- *
+ *
* @param param
*/
void init(String param);
-
-/**
- * Returns the AWS access key ID for this credentials object.
- *
- * @return The AWS access key ID for this credentials object.
- */
-String getAWSAccessKeyId();
/**
- * Returns the AWS secret access key for this credentials object.
+ * Returned {@link AWSCredentialsProvider} can give {@link AWSCredentials}
in case credential belongs to IAM user or
+ * it can return {@link BasicSessionCredentials} if user wants to generate
temporary credential for a given IAM
+ * role.
*
- * @return The AWS secret access key for this credentials object.
- */
-String getAWSSecretKey();
-
-/**
- * Forces this credentials provider to refresh its credentials. For many
- * implementations of credentials provider, this method may simply be a
- * no-op, such as any credentials provider implementation that vends
- * static/non-changing credentials. For other implementations that vend
- * different credentials through out their lifetime, this method should
- * force the credentials provider to refresh its credentials.
+ * @return
*/
-void refresh();
+AWSCredentialsProvider getCredentialProvider();
}
diff --git
a/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java
b/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java
index 39af1f3..5a43312 100644
---
a/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java
+++
b/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java
@@ -30,6 +30,9 @@ import java.nio.ByteBuffer;
import java.util.Map;
import java.util.concurrent.TimeUnit;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang3.builder.ReflectionToStringBuilder;
+import org.apache.commons.lang3.builder.ToStringStyle;
import org.apache.pulsar.io.core.RecordContext;
import org.apache.pulsar.io.core.Sink;
import org.slf4j.Logger;
@@ -37,20 +40,35 @@ import
[GitHub] david-streamlio opened a new issue #1980: Add HDFS-compatible off-loader for data storage
david-streamlio opened a new issue #1980: Add HDFS-compatible off-loader for
data storage
URL: https://github.com/apache/incubator-pulsar/issues/1980
Expected behavior
Add the ability to offload data to HDFS storage, which will be useful for
smart data archiving.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] massakam commented on issue #1977: Enable listener to receive messages even if receiver queue size is zero
massakam commented on issue #1977: Enable listener to receive messages even if receiver queue size is zero URL: https://github.com/apache/incubator-pulsar/pull/1977#issuecomment-398134413 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat closed pull request #1979: Fix typo in bookkeeper.conf
merlimat closed pull request #1979: Fix typo in bookkeeper.conf URL: https://github.com/apache/incubator-pulsar/pull/1979 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/conf/bookkeeper.conf b/conf/bookkeeper.conf index 95620773b8..6742c365bf 100644 --- a/conf/bookkeeper.conf +++ b/conf/bookkeeper.conf @@ -52,7 +52,7 @@ minUsableSizeForIndexFileCreation=1073741824 # Configure a specific hostname or IP address that the bookie should use to advertise itself to # clients. If not set, bookie will advertised its own IP address or hostname, depending on the -# listeningInterface and `seHostNameAsBookieID settings. +# listeningInterface and useHostNameAsBookieID settings. advertisedAddress= # Whether the bookie allowed to use a loopback interface as its primary This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Fix typo in bookkeeper.conf (#1979)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new ee061e2 Fix typo in bookkeeper.conf (#1979) ee061e2 is described below commit ee061e29b5fa5da570d2f11fc773d4afb9f7a22b Author: Ivan Kelly AuthorDate: Mon Jun 18 19:27:09 2018 +0200 Fix typo in bookkeeper.conf (#1979) --- conf/bookkeeper.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf/bookkeeper.conf b/conf/bookkeeper.conf index 9562077..6742c36 100644 --- a/conf/bookkeeper.conf +++ b/conf/bookkeeper.conf @@ -52,7 +52,7 @@ minUsableSizeForIndexFileCreation=1073741824 # Configure a specific hostname or IP address that the bookie should use to advertise itself to # clients. If not set, bookie will advertised its own IP address or hostname, depending on the -# listeningInterface and `seHostNameAsBookieID settings. +# listeningInterface and useHostNameAsBookieID settings. advertisedAddress= # Whether the bookie allowed to use a loopback interface as its primary -- To stop receiving notification emails like this one, please contact mme...@apache.org.
[GitHub] merlimat commented on issue #1972: Add documentation and flexible aws-credential plugin to support aws-role
merlimat commented on issue #1972: Add documentation and flexible aws-credential plugin to support aws-role URL: https://github.com/apache/incubator-pulsar/pull/1972#issuecomment-398127386 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[incubator-pulsar] branch master updated: Addind version 1.22.1 to archived_releases (#1976)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git The following commit(s) were added to refs/heads/master by this push: new 7134783 Addind version 1.22.1 to archived_releases (#1976) 7134783 is described below commit 713478324ea8e5bf6f8dd527f5dc01bdac6663b4 Author: Jai Asher AuthorDate: Mon Jun 18 10:10:03 2018 -0700 Addind version 1.22.1 to archived_releases (#1976) --- site/_config.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/site/_config.yml b/site/_config.yml index 9809ee3..ee337e3 100644 --- a/site/_config.yml +++ b/site/_config.yml @@ -29,6 +29,7 @@ preview_version_id: 20180426.125800-32 current_version: 2.0.0-rc1-incubating python_latest: 2.0.0 archived_releases: + - 1.22.1-incubating - 1.22.0-incubating - 1.21.0-incubating - 1.20.0-incubating -- To stop receiving notification emails like this one, please contact mme...@apache.org.
[GitHub] merlimat closed pull request #1976: Adding version 1.22.1 to archived_releases
merlimat closed pull request #1976: Adding version 1.22.1 to archived_releases URL: https://github.com/apache/incubator-pulsar/pull/1976 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/site/_config.yml b/site/_config.yml index 9809ee3b41..ee337e357e 100644 --- a/site/_config.yml +++ b/site/_config.yml @@ -29,6 +29,7 @@ preview_version_id: 20180426.125800-32 current_version: 2.0.0-rc1-incubating python_latest: 2.0.0 archived_releases: + - 1.22.1-incubating - 1.22.0-incubating - 1.21.0-incubating - 1.20.0-incubating This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ivankelly opened a new pull request #1979: Fix typo in bookkeeper.conf
ivankelly opened a new pull request #1979: Fix typo in bookkeeper.conf URL: https://github.com/apache/incubator-pulsar/pull/1979 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ivankelly closed pull request #1978: Typo in bookkeeper.conf documentation
ivankelly closed pull request #1978: Typo in bookkeeper.conf documentation URL: https://github.com/apache/incubator-pulsar/pull/1978 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ivankelly commented on issue #1978: Typo in bookkeeper.conf documentation
ivankelly commented on issue #1978: Typo in bookkeeper.conf documentation URL: https://github.com/apache/incubator-pulsar/pull/1978#issuecomment-398103627 created on wrong repo. Deleting This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ivankelly opened a new pull request #1978: Typo in bookkeeper.conf documentation
ivankelly opened a new pull request #1978: Typo in bookkeeper.conf documentation URL: https://github.com/apache/incubator-pulsar/pull/1978 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] massakam commented on issue #1977: Enable listener to receive messages even if receiver queue size is zero
massakam commented on issue #1977: Enable listener to receive messages even if receiver queue size is zero URL: https://github.com/apache/incubator-pulsar/pull/1977#issuecomment-398057968 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] massakam commented on issue #1977: Enable listener to receive messages even if receiver queue size is zero
massakam commented on issue #1977: Enable listener to receive messages even if receiver queue size is zero URL: https://github.com/apache/incubator-pulsar/pull/1977#issuecomment-398032895 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] massakam commented on issue #1977: Enable listener to receive messages even if receiver queue size is zero
massakam commented on issue #1977: Enable listener to receive messages even if receiver queue size is zero URL: https://github.com/apache/incubator-pulsar/pull/1977#issuecomment-398057968 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] massakam commented on issue #1977: Enable listener to receive messages even if receiver queue size is zero
massakam commented on issue #1977: Enable listener to receive messages even if receiver queue size is zero URL: https://github.com/apache/incubator-pulsar/pull/1977#issuecomment-398057245 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] massakam commented on issue #1977: Enable listener to receive messages even if receiver queue size is zero
massakam commented on issue #1977: Enable listener to receive messages even if receiver queue size is zero URL: https://github.com/apache/incubator-pulsar/pull/1977#issuecomment-398057245 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] massakam commented on issue #1977: Enable listener to receive messages even if receiver queue size is zero
massakam commented on issue #1977: Enable listener to receive messages even if receiver queue size is zero URL: https://github.com/apache/incubator-pulsar/pull/1977#issuecomment-398032895 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sijie closed pull request #1941: Tiered Storage documentation
sijie closed pull request #1941: Tiered Storage documentation
URL: https://github.com/apache/incubator-pulsar/pull/1941
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/site/_data/cli/pulsar-admin.yaml b/site/_data/cli/pulsar-admin.yaml
index bfae4eadbe..dbbbf794a1 100644
--- a/site/_data/cli/pulsar-admin.yaml
+++ b/site/_data/cli/pulsar-admin.yaml
@@ -469,6 +469,17 @@ commands:
- flags: -t, --threshold
description: "Maximum number of bytes in a topic backlog before
compaction is triggered (eg: 10M, 16G, 3T). 0 disables automatic compaction"
default: '0'
+ - name: get-offload-threshold
+description: Get offloadThreshold for a namespace
+argument: tenant/namespace
+ - name: set-offload-threshold
+description: Set offloadThreshold for a namespace
+argument: tenant/namespace
+options:
+ - flags: -s, --size
+description: "Maximum number of bytes stored in the pulsar cluster for
a topic before data will start being automatically offloaded to longterm
storage (eg: 10M, 16G, 3T, 100). Negative values disable automatic offload. 0
triggers offloading as soon as possible."
+default: '-1'
+
- name: ns-isolation-policy
description: Operations for managing namespace isolation policies.
subcommands:
@@ -512,6 +523,19 @@ commands:
- flags: -w, --wait-complete
description: Wait for compaction to complete
default: 'false'
+ - name: offload
+description: Trigger offload of data from a topic to long-term storage
(e.g. Amazon S3)
+argument: "persistent://tenant/namespace/topic"
+options:
+- flags: -s, --size-threshold
+ description: The maximum amount of data to keep in BookKeeper for the
specific topic
+ - name: offload-status
+description: Check the status of data offloading from a topic to long-term
storage
+argument: "persistent://tenant/namespace/topic"
+options:
+- flags: -w, --wait-complete
+ description: Wait for offloading to complete
+ default: false
- name: create-partitioned-topic
description: Create a partitioned topic. A partitioned topic must be
created before producers can publish to it.
argument: "{persistent|non-persistent}://tenant/namespace/topic"
diff --git a/site/_data/sidebar.yaml b/site/_data/sidebar.yaml
index bf8d5665f2..5516476160 100644
--- a/site/_data/sidebar.yaml
+++ b/site/_data/sidebar.yaml
@@ -134,6 +134,8 @@ groups:
- title: Cookbooks
dir: cookbooks
docs:
+ - title: Tiered Storage
+endpoint: tiered-storage
- title: Topic compaction
endpoint: compaction
- title: Managing message deduplication
diff --git a/site/docs/latest/cookbooks/tiered-storage.md
b/site/docs/latest/cookbooks/tiered-storage.md
new file mode 100644
index 00..4533fa52c2
--- /dev/null
+++ b/site/docs/latest/cookbooks/tiered-storage.md
@@ -0,0 +1,132 @@
+---
+title: Tiered Storage
+tags: [admin, tiered-storage]
+---
+
+Pulsar's **Tiered Storage** feature allows older backlog data to be offloaded
to long term storage, thereby freeing up space in BookKeeper and reducing
storage costs. This cookbook walks you through using tiered storage in your
Pulsar cluster.
+
+## When should I use Tiered Storage?
+
+Tiered storage should be used when you have a topic for which you want to keep
a very long backlog for a long time. For example, if you have a topic
containing user actions which you use to train your recommendation systems, you
may want to keep that data for a long time, so that if you change your
recommendation algorithm you can rerun it against your full user history.
+
+## The offloading mechanism
+
+A topic in Pulsar is backed by a log, known as a managed ledger. This log is
composed of an ordered list of segments. Pulsar only every writes to the final
segment of the log. All previous segments are sealed. The data within the
segment is immutable. This is known as a segment oriented architecture.
+
+{% include figure.html src="/img/pulsar-tiered-storage.png" alt="Tiered
Storage" width="80" %}
+
+The Tiered Storage offloading mechanism takes advantage of this segment
oriented architecture. When offloading is requested, the segments of the log
are copied, one-by-one, to tiered storage. All segments of the log, apart from
the segment currently being written to can be offloaded.
+
+## Amazon S3
+
+Tiered storage currently supports S3 for long term storage. On the broker, the
administrator must configure a S3 bucket and the AWS region where the bucket
exists. Offloaded data will be placed into this bucket.
+
+The configured S3 bucket must exist before attempting to offload. If it does
not exist, the offload operation will fail.
+
+Pulsar users multipart objects to update the segment
[incubator-pulsar] branch master updated: Tiered Storage documentation (#1941)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 6e0afee Tiered Storage documentation (#1941)
6e0afee is described below
commit 6e0afee599b180d177e4fbfa9d7e720efa4a
Author: Ivan Kelly
AuthorDate: Mon Jun 18 09:35:44 2018 +0200
Tiered Storage documentation (#1941)
* Tiered Storage documentation
The patch adds a section in "Concepts and Architecture" and a cookbook
for setting up tiered storage with S3.
Master Issue: #1511
* Review comments, and pulled a couple things in from Luc's wip patch
* Added automatic triggering and addressed review comments
* Add offload threshold commands to reference
---
site/_data/cli/pulsar-admin.yaml | 24
site/_data/sidebar.yaml| 2 +
site/docs/latest/cookbooks/tiered-storage.md | 132 +
.../getting-started/ConceptsAndArchitecture.md | 14 +++
site/img/pulsar-tiered-storage.png | Bin 0 -> 9351 bytes
5 files changed, 172 insertions(+)
diff --git a/site/_data/cli/pulsar-admin.yaml b/site/_data/cli/pulsar-admin.yaml
index bfae4ea..dbbbf79 100644
--- a/site/_data/cli/pulsar-admin.yaml
+++ b/site/_data/cli/pulsar-admin.yaml
@@ -469,6 +469,17 @@ commands:
- flags: -t, --threshold
description: "Maximum number of bytes in a topic backlog before
compaction is triggered (eg: 10M, 16G, 3T). 0 disables automatic compaction"
default: '0'
+ - name: get-offload-threshold
+description: Get offloadThreshold for a namespace
+argument: tenant/namespace
+ - name: set-offload-threshold
+description: Set offloadThreshold for a namespace
+argument: tenant/namespace
+options:
+ - flags: -s, --size
+description: "Maximum number of bytes stored in the pulsar cluster for
a topic before data will start being automatically offloaded to longterm
storage (eg: 10M, 16G, 3T, 100). Negative values disable automatic offload. 0
triggers offloading as soon as possible."
+default: '-1'
+
- name: ns-isolation-policy
description: Operations for managing namespace isolation policies.
subcommands:
@@ -512,6 +523,19 @@ commands:
- flags: -w, --wait-complete
description: Wait for compaction to complete
default: 'false'
+ - name: offload
+description: Trigger offload of data from a topic to long-term storage
(e.g. Amazon S3)
+argument: "persistent://tenant/namespace/topic"
+options:
+- flags: -s, --size-threshold
+ description: The maximum amount of data to keep in BookKeeper for the
specific topic
+ - name: offload-status
+description: Check the status of data offloading from a topic to long-term
storage
+argument: "persistent://tenant/namespace/topic"
+options:
+- flags: -w, --wait-complete
+ description: Wait for offloading to complete
+ default: false
- name: create-partitioned-topic
description: Create a partitioned topic. A partitioned topic must be
created before producers can publish to it.
argument: "{persistent|non-persistent}://tenant/namespace/topic"
diff --git a/site/_data/sidebar.yaml b/site/_data/sidebar.yaml
index 81578ba..59752d0 100644
--- a/site/_data/sidebar.yaml
+++ b/site/_data/sidebar.yaml
@@ -134,6 +134,8 @@ groups:
- title: Cookbooks
dir: cookbooks
docs:
+ - title: Tiered Storage
+endpoint: tiered-storage
- title: Topic compaction
endpoint: compaction
- title: Managing message deduplication
diff --git a/site/docs/latest/cookbooks/tiered-storage.md
b/site/docs/latest/cookbooks/tiered-storage.md
new file mode 100644
index 000..4533fa5
--- /dev/null
+++ b/site/docs/latest/cookbooks/tiered-storage.md
@@ -0,0 +1,132 @@
+---
+title: Tiered Storage
+tags: [admin, tiered-storage]
+---
+
+Pulsar's **Tiered Storage** feature allows older backlog data to be offloaded
to long term storage, thereby freeing up space in BookKeeper and reducing
storage costs. This cookbook walks you through using tiered storage in your
Pulsar cluster.
+
+## When should I use Tiered Storage?
+
+Tiered storage should be used when you have a topic for which you want to keep
a very long backlog for a long time. For example, if you have a topic
containing user actions which you use to train your recommendation systems, you
may want to keep that data for a long time, so that if you change your
recommendation algorithm you can rerun it against your full user history.
+
+## The offloading mechanism
+
+A topic in Pulsar is backed by a log, known as a managed ledger. This log is
composed of an ordered list of segments. Pulsar only every writes to the final
segment of the log. All previous segments are sealed. The data within the
segment is immutable. This is
[incubator-pulsar] branch master updated: Offload threshold policy for namespaces (#1973)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new b2a373d Offload threshold policy for namespaces (#1973)
b2a373d is described below
commit b2a373d903c183c95dd5e29b317ef03af9bf81ff
Author: Ivan Kelly
AuthorDate: Mon Jun 18 09:35:13 2018 +0200
Offload threshold policy for namespaces (#1973)
Allow administrators to specify a offload threshold policy on a
namespace, which stipulates that once a topic in the namespace has a
certain amount of data on the pulsar cluster, start offloading some of
this data to longterm storage.
This patch also cleans up TestS3Offload, and adds offload status each
ledger in topic internal stats.
Master Issue: #1511
---
.../pulsar/broker/admin/impl/NamespacesBase.java | 37 ++
.../apache/pulsar/broker/admin/v1/Namespaces.java | 31 +
.../apache/pulsar/broker/admin/v2/Namespaces.java | 29 +
.../pulsar/broker/service/BrokerService.java | 2 +
.../broker/service/persistent/PersistentTopic.java | 1 +
.../org/apache/pulsar/client/admin/Namespaces.java | 48
.../client/admin/internal/NamespacesImpl.java | 22
.../org/apache/pulsar/admin/cli/CmdNamespaces.java | 36 ++
.../data/PersistentTopicInternalStats.java | 1 +
.../pulsar/common/policies/data/Policies.java | 7 +-
.../pulsar/tests/integration/TestS3Offload.java| 134 -
11 files changed, 313 insertions(+), 35 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
index bba3dc7..b8a4c53 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
@@ -1570,5 +1570,42 @@ public abstract class NamespacesBase extends
AdminResource {
}
}
+protected long internalGetOffloadThreshold() {
+validateAdminAccessForTenant(namespaceName.getTenant());
+return getNamespacePolicies(namespaceName).offload_threshold;
+}
+
+protected void internalSetOffloadThreshold(long newThreshold) {
+validateSuperUserAccess();
+validatePoliciesReadOnlyAccess();
+
+try {
+Stat nodeStat = new Stat();
+final String path = path(POLICIES, namespaceName.toString());
+byte[] content = globalZk().getData(path, null, nodeStat);
+Policies policies = jsonMapper().readValue(content,
Policies.class);
+policies.offload_threshold = newThreshold;
+globalZk().setData(path, jsonMapper().writeValueAsBytes(policies),
nodeStat.getVersion());
+policiesCache().invalidate(path(POLICIES,
namespaceName.toString()));
+log.info("[{}] Successfully updated offloadThreshold
configuration: namespace={}, value={}",
+ clientAppId(), namespaceName,
policies.compaction_threshold);
+
+} catch (KeeperException.NoNodeException e) {
+log.warn("[{}] Failed to update offloadThreshold configuration for
namespace {}: does not exist",
+ clientAppId(), namespaceName);
+throw new RestException(Status.NOT_FOUND, "Namespace does not
exist");
+} catch (KeeperException.BadVersionException e) {
+log.warn("[{}] Failed to update offloadThreshold configuration for
namespace {}: concurrent modification",
+ clientAppId(), namespaceName);
+throw new RestException(Status.CONFLICT, "Concurrent
modification");
+} catch (RestException pfe) {
+throw pfe;
+} catch (Exception e) {
+log.error("[{}] Failed to update offloadThreshold configuration
for namespace {}",
+ clientAppId(), namespaceName, e);
+throw new RestException(e);
+}
+}
+
private static final Logger log =
LoggerFactory.getLogger(NamespacesBase.class);
}
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/Namespaces.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/Namespaces.java
index f8edcf1..bbc4438 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/Namespaces.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/Namespaces.java
@@ -746,5 +746,36 @@ public class Namespaces extends NamespacesBase {
internalSetCompactionThreshold(newThreshold);
}
+@GET
+@Path("/{property}/{cluster}/{namespace}/offloadThreshold")
+@ApiOperation(value = "Maximum number of bytes stored on the pulsar
cluster for a topic,"
+ + " before the
[GitHub] sijie closed pull request #1973: Offload threshold policy for namespaces
sijie closed pull request #1973: Offload threshold policy for namespaces
URL: https://github.com/apache/incubator-pulsar/pull/1973
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
index bba3dc7ef2..b8a4c53a38 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
@@ -1570,5 +1570,42 @@ protected void internalSetCompactionThreshold(long
newThreshold) {
}
}
+protected long internalGetOffloadThreshold() {
+validateAdminAccessForTenant(namespaceName.getTenant());
+return getNamespacePolicies(namespaceName).offload_threshold;
+}
+
+protected void internalSetOffloadThreshold(long newThreshold) {
+validateSuperUserAccess();
+validatePoliciesReadOnlyAccess();
+
+try {
+Stat nodeStat = new Stat();
+final String path = path(POLICIES, namespaceName.toString());
+byte[] content = globalZk().getData(path, null, nodeStat);
+Policies policies = jsonMapper().readValue(content,
Policies.class);
+policies.offload_threshold = newThreshold;
+globalZk().setData(path, jsonMapper().writeValueAsBytes(policies),
nodeStat.getVersion());
+policiesCache().invalidate(path(POLICIES,
namespaceName.toString()));
+log.info("[{}] Successfully updated offloadThreshold
configuration: namespace={}, value={}",
+ clientAppId(), namespaceName,
policies.compaction_threshold);
+
+} catch (KeeperException.NoNodeException e) {
+log.warn("[{}] Failed to update offloadThreshold configuration for
namespace {}: does not exist",
+ clientAppId(), namespaceName);
+throw new RestException(Status.NOT_FOUND, "Namespace does not
exist");
+} catch (KeeperException.BadVersionException e) {
+log.warn("[{}] Failed to update offloadThreshold configuration for
namespace {}: concurrent modification",
+ clientAppId(), namespaceName);
+throw new RestException(Status.CONFLICT, "Concurrent
modification");
+} catch (RestException pfe) {
+throw pfe;
+} catch (Exception e) {
+log.error("[{}] Failed to update offloadThreshold configuration
for namespace {}",
+ clientAppId(), namespaceName, e);
+throw new RestException(e);
+}
+}
+
private static final Logger log =
LoggerFactory.getLogger(NamespacesBase.class);
}
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/Namespaces.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/Namespaces.java
index f8edcf126c..bbc4438626 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/Namespaces.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v1/Namespaces.java
@@ -746,5 +746,36 @@ public void setCompactionThreshold(@PathParam("property")
String property,
internalSetCompactionThreshold(newThreshold);
}
+@GET
+@Path("/{property}/{cluster}/{namespace}/offloadThreshold")
+@ApiOperation(value = "Maximum number of bytes stored on the pulsar
cluster for a topic,"
+ + " before the broker will start offloading to
longterm storage",
+ notes = "A negative value disables automatic offloading")
+@ApiResponses(value = { @ApiResponse(code = 403, message = "Don't have
admin permission"),
+@ApiResponse(code = 404, message = "Namespace
doesn't exist") })
+public long getOffloadThreshold(@PathParam("property") String property,
+@PathParam("cluster") String cluster,
+@PathParam("namespace") String namespace) {
+validateNamespaceName(property, cluster, namespace);
+return internalGetOffloadThreshold();
+}
+
+@PUT
+@Path("/{property}/{cluster}/{namespace}/offloadThreshold")
+@ApiOperation(value = "Set maximum number of bytes stored on the pulsar
cluster for a topic,"
+ + " before the broker will start offloading to
longterm storage",
+ notes = "A negative value disables automatic offloading")
+@ApiResponses(value = { @ApiResponse(code = 403, message = "Don't have
admin permission"),
+@ApiResponse(code = 404, message = "Namespace
doesn't exist"),
+
[incubator-pulsar] branch master updated: Improved instruction for getting started with C++ client lib (#1975)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new f221b26 Improved instruction for getting started with C++ client lib
(#1975)
f221b26 is described below
commit f221b2652e51e5a539da0431cb27d2520e748a7e
Author: Matteo Merli
AuthorDate: Mon Jun 18 00:34:42 2018 -0700
Improved instruction for getting started with C++ client lib (#1975)
### Motivation
Documentation for C++ library was mixed with instructions on how to build
the library and how to use it.
### Modifications
* Split the C++ client page in 2 parts, one for building and one for using
* Added instructions to build RPM, which is much easier than build
directly (since it builds in Docker with all dependencies installed).
---
site/_data/sidebar.yaml| 2 +
site/docs/latest/clients/Cpp.md| 128 +++--
.../{clients/Cpp.md => project/CompileCpp.md} | 76 +---
3 files changed, 44 insertions(+), 162 deletions(-)
diff --git a/site/_data/sidebar.yaml b/site/_data/sidebar.yaml
index bf8d566..81578ba 100644
--- a/site/_data/sidebar.yaml
+++ b/site/_data/sidebar.yaml
@@ -162,6 +162,8 @@ groups:
endpoint: schema-storage
- title: Modular load manager
endpoint: ModularLoadManager
+ - title: Building Pulsar C++ client
+endpoint: CompileCpp
- title: Reference
dir: reference
diff --git a/site/docs/latest/clients/Cpp.md b/site/docs/latest/clients/Cpp.md
index d153be9..34f36ed 100644
--- a/site/docs/latest/clients/Cpp.md
+++ b/site/docs/latest/clients/Cpp.md
@@ -24,106 +24,64 @@ tags: [client, cpp]
-->
-
-
-{% include admonition.html type='info' content="
-We welcome contributions from the open source community, kindly make sure your
changes are backward compatible with gcc-4.4.7 and Boost 1.41.
-" %}
-
## Supported platforms
The Pulsar C++ client has been successfully tested on **MacOS** and **Linux**.
-## System requirements
-
-You need to have the following installed to use the C++ client:
-
-* [CMake](https://cmake.org/)
-* [Boost](http://www.boost.org/)
-* [Protocol Buffers](https://developers.google.com/protocol-buffers/) 2.6
-* [Log4CXX](https://logging.apache.org/log4cxx)
-* [libcurl](https://curl.haxx.se/libcurl/)
-* [Google Test](https://github.com/google/googletest)
-* [JsonCpp](https://github.com/open-source-parsers/jsoncpp)
+## Linux
-## Compilation
+There are recipes that build RPM and Debian packages containing a
+statically linked `libpulsar.so` / `libpulsar.a` with all the required
+dependencies.
-There are separate compilation instructions for [MacOS](#macos) and
[Linux](#linux). For both systems, start by cloning the Pulsar repository:
+To build the C++ library packages, first build the Java packages:
```shell
-$ git clone {{ site.pulsar_repo }}
+mvn install -DskipTests
```
-### Linux
-
-First, install all of the necessary dependencies:
+ RPM
```shell
-$ apt-get install cmake libssl-dev libcurl4-openssl-dev liblog4cxx-dev \
- libprotobuf-dev libboost-all-dev google-mock libgtest-dev libjsoncpp-dev
+pulsar-client-cpp/pkg/rpm/docker-build-rpm.sh
```
-Then compile and install [Google Test](https://github.com/google/googletest):
+This will build the RPM inside a Docker container and it will leave the RPMs
+in `pulsar-client-cpp/pkg/rpm/RPMS/x86_64/`.
-```shell
-# libgtest-dev version is 1.18.0 or above
-$ cd /usr/src/googletest
-$ sudo cmake .
-$ sudo make
-$ sudo cp ./googlemock/libgmock.a ./googletest/libgtest.a /usr/lib/
-
-# less than 1.18.0
-$ cd /usr/src/gtest
-$ sudo cmake .
-$ sudo make
-$ sudo cp libgtest.a /usr/lib
-
-$ cd /usr/src/gmock
-$ sudo cmake .
-$ sudo make
-$ sudo cp libgmock.a /usr/lib
-```
+| Package name | Content |
+|-|-|
+| pulsar-client | Shared library `libpulsar.so` |
+| pulsar-client-devel | Static library `libpulsar.a` and C++ and C headers |
+| pulsar-client-debuginfo | Debug symbols for `libpulsar.so` |
-Finally, compile the Pulsar client library for C++ inside the Pulsar repo:
+ Deb
+
+To build Debian packages:
```shell
-$ cd pulsar-client-cpp
-$ cmake .
-$ make
+pulsar-client-cpp/pkg/deb/docker-build-deb.sh
```
-The resulting files, `libpulsar.so` and `libpulsar.a`, will be placed in the
`lib` folder of the repo while two tools, `perfProducer` and `perfConsumer`,
will be placed in the `perf` directory.
+Debian packages will be created at `pulsar-client-cpp/pkg/deb/BUILD/DEB/`
+
+| Package name | Content |
+|-|-|
+| pulsar-client | Shared library `libpulsar.so` |
+| pulsar-client-dev | Static library `libpulsar.a` and C++ and C headers |
-### MacOS
+## MacOS
-First, install all of the necessary dependencies:
+Use the [Homebrew](https://brew.sh/) supplied recipe to build the Pulsar
+client lib
[GitHub] sijie closed pull request #1975: Improved instruction for getting started with C++ client lib
sijie closed pull request #1975: Improved instruction for getting started with
C++ client lib
URL: https://github.com/apache/incubator-pulsar/pull/1975
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/site/_data/sidebar.yaml b/site/_data/sidebar.yaml
index bf8d5665f2..81578bad68 100644
--- a/site/_data/sidebar.yaml
+++ b/site/_data/sidebar.yaml
@@ -162,6 +162,8 @@ groups:
endpoint: schema-storage
- title: Modular load manager
endpoint: ModularLoadManager
+ - title: Building Pulsar C++ client
+endpoint: CompileCpp
- title: Reference
dir: reference
diff --git a/site/docs/latest/clients/Cpp.md b/site/docs/latest/clients/Cpp.md
index d153be92e6..34f36edd03 100644
--- a/site/docs/latest/clients/Cpp.md
+++ b/site/docs/latest/clients/Cpp.md
@@ -24,106 +24,64 @@ tags: [client, cpp]
-->
-
-
-{% include admonition.html type='info' content="
-We welcome contributions from the open source community, kindly make sure your
changes are backward compatible with gcc-4.4.7 and Boost 1.41.
-" %}
-
## Supported platforms
The Pulsar C++ client has been successfully tested on **MacOS** and **Linux**.
-## System requirements
-
-You need to have the following installed to use the C++ client:
-
-* [CMake](https://cmake.org/)
-* [Boost](http://www.boost.org/)
-* [Protocol Buffers](https://developers.google.com/protocol-buffers/) 2.6
-* [Log4CXX](https://logging.apache.org/log4cxx)
-* [libcurl](https://curl.haxx.se/libcurl/)
-* [Google Test](https://github.com/google/googletest)
-* [JsonCpp](https://github.com/open-source-parsers/jsoncpp)
+## Linux
-## Compilation
+There are recipes that build RPM and Debian packages containing a
+statically linked `libpulsar.so` / `libpulsar.a` with all the required
+dependencies.
-There are separate compilation instructions for [MacOS](#macos) and
[Linux](#linux). For both systems, start by cloning the Pulsar repository:
+To build the C++ library packages, first build the Java packages:
```shell
-$ git clone {{ site.pulsar_repo }}
+mvn install -DskipTests
```
-### Linux
-
-First, install all of the necessary dependencies:
+ RPM
```shell
-$ apt-get install cmake libssl-dev libcurl4-openssl-dev liblog4cxx-dev \
- libprotobuf-dev libboost-all-dev google-mock libgtest-dev libjsoncpp-dev
+pulsar-client-cpp/pkg/rpm/docker-build-rpm.sh
```
-Then compile and install [Google Test](https://github.com/google/googletest):
+This will build the RPM inside a Docker container and it will leave the RPMs
+in `pulsar-client-cpp/pkg/rpm/RPMS/x86_64/`.
-```shell
-# libgtest-dev version is 1.18.0 or above
-$ cd /usr/src/googletest
-$ sudo cmake .
-$ sudo make
-$ sudo cp ./googlemock/libgmock.a ./googletest/libgtest.a /usr/lib/
-
-# less than 1.18.0
-$ cd /usr/src/gtest
-$ sudo cmake .
-$ sudo make
-$ sudo cp libgtest.a /usr/lib
-
-$ cd /usr/src/gmock
-$ sudo cmake .
-$ sudo make
-$ sudo cp libgmock.a /usr/lib
-```
+| Package name | Content |
+|-|-|
+| pulsar-client | Shared library `libpulsar.so` |
+| pulsar-client-devel | Static library `libpulsar.a` and C++ and C headers |
+| pulsar-client-debuginfo | Debug symbols for `libpulsar.so` |
-Finally, compile the Pulsar client library for C++ inside the Pulsar repo:
+ Deb
+
+To build Debian packages:
```shell
-$ cd pulsar-client-cpp
-$ cmake .
-$ make
+pulsar-client-cpp/pkg/deb/docker-build-deb.sh
```
-The resulting files, `libpulsar.so` and `libpulsar.a`, will be placed in the
`lib` folder of the repo while two tools, `perfProducer` and `perfConsumer`,
will be placed in the `perf` directory.
+Debian packages will be created at `pulsar-client-cpp/pkg/deb/BUILD/DEB/`
+
+| Package name | Content |
+|-|-|
+| pulsar-client | Shared library `libpulsar.so` |
+| pulsar-client-dev | Static library `libpulsar.a` and C++ and C headers |
-### MacOS
+## MacOS
-First, install all of the necessary dependencies:
+Use the [Homebrew](https://brew.sh/) supplied recipe to build the Pulsar
+client lib on MacOS.
```shell
-# OpenSSL installation
-$ brew install openssl
-$ export OPENSSL_INCLUDE_DIR=/usr/local/opt/openssl/include/
-$ export OPENSSL_ROOT_DIR=/usr/local/opt/openssl/
-
-# Protocol Buffers installation
-$ brew tap homebrew/versions
-$ brew install protobuf260
-$ brew install boost
-$ brew install log4cxx
-
-# Google Test installation
-$ git clone https://github.com/google/googletest.git
-$ cd googletest
-$ cmake .
-$ make install
+brew install
https://raw.githubusercontent.com/apache/incubator-pulsar/master/pulsar-client-cpp/homebrew/libpulsar.rb
```
-Then compile the Pulsar client library in the repo that you cloned:
+If using Python 3 on MacOS, add the flag `--with-python3` to the above command.
-```shell
-$ cd pulsar-client-cpp
-$ cmake .
[GitHub] rdhabalia commented on issue #1972: Add documentation and flexible aws-credential plugin to support aws-role
rdhabalia commented on issue #1972: Add documentation and flexible aws-credential plugin to support aws-role URL: https://github.com/apache/incubator-pulsar/pull/1972#issuecomment-397958436 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia commented on issue #1972: Add documentation and flexible aws-credential plugin to support aws-role
rdhabalia commented on issue #1972: Add documentation and flexible aws-credential plugin to support aws-role URL: https://github.com/apache/incubator-pulsar/pull/1972#issuecomment-397958532 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia commented on issue #1972: Add documentation and flexible aws-credential plugin to support aws-role
rdhabalia commented on issue #1972: Add documentation and flexible aws-credential plugin to support aws-role URL: https://github.com/apache/incubator-pulsar/pull/1972#issuecomment-397958532 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] rdhabalia commented on issue #1972: Add documentation and flexible aws-credential plugin to support aws-role
rdhabalia commented on issue #1972: Add documentation and flexible aws-credential plugin to support aws-role URL: https://github.com/apache/incubator-pulsar/pull/1972#issuecomment-397958436 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] massakam opened a new pull request #1977: Enable listener to receive messages even if receiver queue size is zero
massakam opened a new pull request #1977: Enable listener to receive messages even if receiver queue size is zero URL: https://github.com/apache/incubator-pulsar/pull/1977 ### Motivation If `receiverQueueSize` is set to 0, consumer using message listener can not receive any messages. I think there are two causes: - The consumer does not send flow command when the connection has been opened - Even if the client receive a message, the message is not added to `incomingMessages` so the listener can not handle it ### Modifications If the queue size is zero and the message listener is registered, - the client sends a flow command when the connection has been opened - the received message is passed directly to the listener without being added to `incomingMessages` I am not confident about this change, so please review carefully. ### Result Even if the queue size is zero, the message listener can process the received messages. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jai1 opened a new pull request #1976: Addind version 1.22.1 to archived_releases
jai1 opened a new pull request #1976: Addind version 1.22.1 to archived_releases URL: https://github.com/apache/incubator-pulsar/pull/1976 ### Motivation Explain here the context, and why you're making that change. What is the problem you're trying to solve. ### Modifications Describe the modifications you've done. ### Result After your change, what will change. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
