date:20180716

[GitHub] zhaijack commented on issue #2065: GCS offload support(2): replace `s3client` api with `jclouds` related api

2018-07-16 Thread GitBox

zhaijack commented on issue #2065: GCS offload support(2): replace `s3client` 
api with `jclouds` related api 
URL: https://github.com/apache/incubator-pulsar/pull/2065#issuecomment-405471599
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat closed issue #2119: TopicPatternSubscription doesn't work through proxy

2018-07-16 Thread GitBox

merlimat closed issue #2119: TopicPatternSubscription doesn't work through proxy
URL: https://github.com/apache/incubator-pulsar/issues/2119
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat closed pull request #2176: Issue 2119: TopicPatternSubscription doesn't work through proxy

2018-07-16 Thread GitBox

merlimat closed pull request #2176: Issue 2119: TopicPatternSubscription 
doesn't work through proxy
URL: https://github.com/apache/incubator-pulsar/pull/2176
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
 
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
index bed7ed7a84..957cff8d93 100644
--- 
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
+++ 
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
@@ -26,6 +26,7 @@
 import java.net.URISyntaxException;
 
 import org.apache.pulsar.common.api.Commands;
+import 
org.apache.pulsar.common.api.proto.PulsarApi.CommandGetTopicsOfNamespace;
 import org.apache.pulsar.common.api.proto.PulsarApi.CommandLookupTopic;
 import 
org.apache.pulsar.common.api.proto.PulsarApi.CommandLookupTopicResponse.LookupType;
 import 
org.apache.pulsar.common.api.proto.PulsarApi.CommandPartitionedTopicMetadata;
@@ -54,6 +55,11 @@
 .build("pulsar_proxy_partitions_metadata_requests", "Counter of 
partitions metadata requests").create()
 .register();
 
+private static final Counter getTopicsOfNamespaceRequestss = Counter
+.build("pulsar_proxy_get_topics_of_namespace_requests", "Counter 
of getTopicsOfNamespace requests")
+.create()
+.register();
+
 static final Counter rejectedLookupRequests = 
Counter.build("pulsar_proxy_rejected_lookup_requests",
 "Counter of topic lookup requests rejected due to 
throttling").create().register();
 
@@ -62,6 +68,11 @@
 "Counter of partitions metadata requests rejected due to 
throttling")
 .create().register();
 
+static final Counter rejectedGetTopicsOfNamespaceRequests = Counter
+.build("pulsar_proxy_rejected_get_topics_of_namespace_requests",
+"Counter of getTopicsOfNamespace requests rejected due to 
throttling")
+.create().register();
+
 public LookupProxyHandler(ProxyService proxy, ProxyConnection 
proxyConnection) {
 this.service = proxy;
 this.proxyConnection = proxyConnection;
@@ -246,5 +257,98 @@ private void 
handlePartitionMetadataResponse(CommandPartitionedTopicMetadata par
 }
 }
 
+public void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace 
commandGetTopicsOfNamespace) {
+getTopicsOfNamespaceRequestss.inc();
+if (log.isDebugEnabled()) {
+log.debug("[{}] Received GetTopicsOfNamespace", clientAddress);
+}
+
+final long requestId = commandGetTopicsOfNamespace.getRequestId();
+
+if (this.service.getLookupRequestSemaphore().tryAcquire()) {
+handleGetTopicsOfNamespace(commandGetTopicsOfNamespace, requestId);
+this.service.getLookupRequestSemaphore().release();
+} else {
+rejectedGetTopicsOfNamespaceRequests.inc();
+if (log.isDebugEnabled()) {
+log.debug("GetTopicsOfNamespace Request ID {} from {} rejected 
- {}.", requestId, clientAddress,
+throttlingErrorMessage);
+}
+proxyConnection.ctx().writeAndFlush(Commands.newError(
+requestId, ServerError.ServiceNotReady, throttlingErrorMessage
+));
+}
+}
+
+
+private void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace 
commandGetTopicsOfNamespace,
+long clientRequestId) {
+String serviceUrl;
+if (isBlank(brokerServiceURL)) {
+ServiceLookupData availableBroker;
+try {
+availableBroker = service.getDiscoveryProvider().nextBroker();
+} catch (Exception e) {
+log.warn("[{}] Failed to get next active broker {}", 
clientAddress, e.getMessage(), e);
+proxyConnection.ctx().writeAndFlush(Commands.newError(
+clientRequestId, ServerError.ServiceNotReady, 
e.getMessage()
+));
+return;
+}
+serviceUrl = this.connectWithTLS ?
+availableBroker.getPulsarServiceUrlTls() : 
availableBroker.getPulsarServiceUrl();
+} else {
+serviceUrl = this.connectWithTLS ?
+service.getConfiguration().getBrokerServiceURLTLS() : 
service.getConfiguration().getBrokerServiceURL();
+}
+performGetTopicsOfNamespace(clientRequestId, 
commandGetTopicsOfNamespace.getNamespace(), serviceUrl, 10);
+}
+
+private void performGetTopicsOfNamespace(long clientRequestId,
+ String namesp

[GitHub] merlimat commented on issue #2173: Removed GPL deps on gnu crypto from aerospike connector

2018-07-16 Thread GitBox

merlimat commented on issue #2173: Removed GPL deps on gnu crypto from 
aerospike connector
URL: https://github.com/apache/incubator-pulsar/pull/2173#issuecomment-405458792
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Issue 2119: TopicPatternSubscription doesn't work through proxy (#2176)

2018-07-16 Thread mmerli

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 5b72eec  Issue 2119: TopicPatternSubscription doesn't work through 
proxy (#2176)
5b72eec is described below

commit 5b72eecb57b56e42f1159ac228de0bc17887a869
Author: Sijie Guo 
AuthorDate: Mon Jul 16 21:51:59 2018 -0700

Issue 2119: TopicPatternSubscription doesn't work through proxy (#2176)

* Fix regex

* Implement GetTopicsOfNamespace on Proxy

* remove unneeded import
---
 .../pulsar/proxy/server/LookupProxyHandler.java| 104 +
 .../pulsar/proxy/server/ProxyConnection.java   |   8 ++
 .../org/apache/pulsar/proxy/server/ProxyTest.java  |  49 ++
 3 files changed, 161 insertions(+)

diff --git 
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
 
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
index bed7ed7..957cff8 100644
--- 
a/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
+++ 
b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/LookupProxyHandler.java
@@ -26,6 +26,7 @@ import java.net.URI;
 import java.net.URISyntaxException;
 
 import org.apache.pulsar.common.api.Commands;
+import 
org.apache.pulsar.common.api.proto.PulsarApi.CommandGetTopicsOfNamespace;
 import org.apache.pulsar.common.api.proto.PulsarApi.CommandLookupTopic;
 import 
org.apache.pulsar.common.api.proto.PulsarApi.CommandLookupTopicResponse.LookupType;
 import 
org.apache.pulsar.common.api.proto.PulsarApi.CommandPartitionedTopicMetadata;
@@ -54,6 +55,11 @@ public class LookupProxyHandler {
 .build("pulsar_proxy_partitions_metadata_requests", "Counter of 
partitions metadata requests").create()
 .register();
 
+private static final Counter getTopicsOfNamespaceRequestss = Counter
+.build("pulsar_proxy_get_topics_of_namespace_requests", "Counter 
of getTopicsOfNamespace requests")
+.create()
+.register();
+
 static final Counter rejectedLookupRequests = 
Counter.build("pulsar_proxy_rejected_lookup_requests",
 "Counter of topic lookup requests rejected due to 
throttling").create().register();
 
@@ -62,6 +68,11 @@ public class LookupProxyHandler {
 "Counter of partitions metadata requests rejected due to 
throttling")
 .create().register();
 
+static final Counter rejectedGetTopicsOfNamespaceRequests = Counter
+.build("pulsar_proxy_rejected_get_topics_of_namespace_requests",
+"Counter of getTopicsOfNamespace requests rejected due to 
throttling")
+.create().register();
+
 public LookupProxyHandler(ProxyService proxy, ProxyConnection 
proxyConnection) {
 this.service = proxy;
 this.proxyConnection = proxyConnection;
@@ -246,5 +257,98 @@ public class LookupProxyHandler {
 }
 }
 
+public void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace 
commandGetTopicsOfNamespace) {
+getTopicsOfNamespaceRequestss.inc();
+if (log.isDebugEnabled()) {
+log.debug("[{}] Received GetTopicsOfNamespace", clientAddress);
+}
+
+final long requestId = commandGetTopicsOfNamespace.getRequestId();
+
+if (this.service.getLookupRequestSemaphore().tryAcquire()) {
+handleGetTopicsOfNamespace(commandGetTopicsOfNamespace, requestId);
+this.service.getLookupRequestSemaphore().release();
+} else {
+rejectedGetTopicsOfNamespaceRequests.inc();
+if (log.isDebugEnabled()) {
+log.debug("GetTopicsOfNamespace Request ID {} from {} rejected 
- {}.", requestId, clientAddress,
+throttlingErrorMessage);
+}
+proxyConnection.ctx().writeAndFlush(Commands.newError(
+requestId, ServerError.ServiceNotReady, throttlingErrorMessage
+));
+}
+}
+
+
+private void handleGetTopicsOfNamespace(CommandGetTopicsOfNamespace 
commandGetTopicsOfNamespace,
+long clientRequestId) {
+String serviceUrl;
+if (isBlank(brokerServiceURL)) {
+ServiceLookupData availableBroker;
+try {
+availableBroker = service.getDiscoveryProvider().nextBroker();
+} catch (Exception e) {
+log.warn("[{}] Failed to get next active broker {}", 
clientAddress, e.getMessage(), e);
+proxyConnection.ctx().writeAndFlush(Commands.newError(
+clientRequestId, ServerError.ServiceNotReady, 
e.getMessage()
+));
+return;
+}
+serviceUrl = this.connectWithTLS ?
+availableBroker.getP

[GitHub] merlimat closed pull request #2172: If auth is badly configured in c++, close the connection

2018-07-16 Thread GitBox

merlimat closed pull request #2172: If auth is badly configured in c++, close 
the connection
URL: https://github.com/apache/incubator-pulsar/pull/2172
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/pulsar-client-cpp/lib/ClientConnection.cc 
b/pulsar-client-cpp/lib/ClientConnection.cc
index 8d8243cd6a..4e6d0f23b6 100644
--- a/pulsar-client-cpp/lib/ClientConnection.cc
+++ b/pulsar-client-cpp/lib/ClientConnection.cc
@@ -160,6 +160,12 @@ ClientConnection::ClientConnection(const std::string& 
logicalAddress, const std:
 }
 }
 
+if (!authentication_) {
+LOG_ERROR("Invalid authentication plugin");
+close();
+return;
+}
+
 AuthenticationDataPtr authData;
 if (authentication_->getAuthData(authData) == ResultOk && 
authData->hasDataForTls()) {
 std::string tlsCertificates = authData->getTlsCertificates();
@@ -354,6 +360,10 @@ void ClientConnection::handleSentPulsarConnect(const 
boost::system::error_code&
  *
  */
 void ClientConnection::tcpConnectAsync() {
+if (isClosed()) {
+return;
+}
+
 boost::system::error_code err;
 Url service_url;
 if (!Url::parse(physicalAddress_, service_url)) {
diff --git a/pulsar-client-cpp/python/pulsar_test.py 
b/pulsar-client-cpp/python/pulsar_test.py
index 3a5c407151..200a107d76 100755
--- a/pulsar-client-cpp/python/pulsar_test.py
+++ b/pulsar-client-cpp/python/pulsar_test.py
@@ -24,7 +24,7 @@
 import os
 from pulsar import Client, MessageId, \
 CompressionType, ConsumerType, PartitionsRoutingMode, \
-AuthenticationTLS
+AuthenticationTLS, Authentication
 
 from _pulsar import ProducerConfiguration, ConsumerConfiguration
 
@@ -152,6 +152,23 @@ def test_tls_auth(self):
 
 client.close()
 
+def test_auth_junk_params(self):
+certs_dir = 
'/pulsar/pulsar-broker/src/test/resources/authentication/tls/'
+if not os.path.exists(certs_dir):
+certs_dir = 
"../../pulsar-broker/src/test/resources/authentication/tls/"
+authPlugin = "someoldjunk.so"
+authParams = "blah"
+client = Client(self.serviceUrlTls,
+tls_trust_certs_file_path=certs_dir + 'cacert.pem',
+tls_allow_insecure_connection=False,
+authentication=Authentication(authPlugin, authParams))
+try:
+
client.subscribe('persistent://property/cluster/namespace/my-python-topic-producer-consumer',
+ 'my-sub',
+ consumer_type=ConsumerType.Shared)
+except:
+pass  # Exception is expected
+
 def test_message_listener(self):
 client = Client(self.serviceUrl)
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: If auth is badly configured in c++, close the connection (#2172)

2018-07-16 Thread mmerli

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new da5b3cd  If auth is badly configured in c++, close the connection 
(#2172)
da5b3cd is described below

commit da5b3cdf1e6fabbcadf6be4adff7c055e37674f5
Author: Ivan Kelly 
AuthorDate: Tue Jul 17 05:51:00 2018 +0100

If auth is badly configured in c++, close the connection (#2172)

* If auth is badly configured in c++, close the connection

This patch fixes a segfault where, if auth was badly configured,
resulting in a null authentication pointer, the client would still try
to use it to connect, and as a result, segfault.

The patch adds a null check on the authentication object, and a check
that the connection object is still valid before initialization of the
tcp channel.

* formatting errors
---
 pulsar-client-cpp/lib/ClientConnection.cc | 10 ++
 pulsar-client-cpp/python/pulsar_test.py   | 19 ++-
 2 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/pulsar-client-cpp/lib/ClientConnection.cc 
b/pulsar-client-cpp/lib/ClientConnection.cc
index 8d8243c..4e6d0f2 100644
--- a/pulsar-client-cpp/lib/ClientConnection.cc
+++ b/pulsar-client-cpp/lib/ClientConnection.cc
@@ -160,6 +160,12 @@ ClientConnection::ClientConnection(const std::string& 
logicalAddress, const std:
 }
 }
 
+if (!authentication_) {
+LOG_ERROR("Invalid authentication plugin");
+close();
+return;
+}
+
 AuthenticationDataPtr authData;
 if (authentication_->getAuthData(authData) == ResultOk && 
authData->hasDataForTls()) {
 std::string tlsCertificates = authData->getTlsCertificates();
@@ -354,6 +360,10 @@ void ClientConnection::handleSentPulsarConnect(const 
boost::system::error_code&
  *
  */
 void ClientConnection::tcpConnectAsync() {
+if (isClosed()) {
+return;
+}
+
 boost::system::error_code err;
 Url service_url;
 if (!Url::parse(physicalAddress_, service_url)) {
diff --git a/pulsar-client-cpp/python/pulsar_test.py 
b/pulsar-client-cpp/python/pulsar_test.py
index 3a5c407..200a107 100755
--- a/pulsar-client-cpp/python/pulsar_test.py
+++ b/pulsar-client-cpp/python/pulsar_test.py
@@ -24,7 +24,7 @@ import time
 import os
 from pulsar import Client, MessageId, \
 CompressionType, ConsumerType, PartitionsRoutingMode, \
-AuthenticationTLS
+AuthenticationTLS, Authentication
 
 from _pulsar import ProducerConfiguration, ConsumerConfiguration
 
@@ -152,6 +152,23 @@ class PulsarTest(TestCase):
 
 client.close()
 
+def test_auth_junk_params(self):
+certs_dir = 
'/pulsar/pulsar-broker/src/test/resources/authentication/tls/'
+if not os.path.exists(certs_dir):
+certs_dir = 
"../../pulsar-broker/src/test/resources/authentication/tls/"
+authPlugin = "someoldjunk.so"
+authParams = "blah"
+client = Client(self.serviceUrlTls,
+tls_trust_certs_file_path=certs_dir + 'cacert.pem',
+tls_allow_insecure_connection=False,
+authentication=Authentication(authPlugin, authParams))
+try:
+
client.subscribe('persistent://property/cluster/namespace/my-python-topic-producer-consumer',
+ 'my-sub',
+ consumer_type=ConsumerType.Shared)
+except:
+pass  # Exception is expected
+
 def test_message_listener(self):
 client = Client(self.serviceUrl)

[GitHub] djKooks commented on issue #2166: 'java.lang.NoSuchMethodError' when running standalone

2018-07-16 Thread GitBox

djKooks commented on issue #2166: 'java.lang.NoSuchMethodError' when running 
standalone
URL: 
https://github.com/apache/incubator-pulsar/issues/2166#issuecomment-405429524
 
 
   @srkukarni @merlimat thanks! I'll try it and let you know.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #2166: 'java.lang.NoSuchMethodError' when running standalone

2018-07-16 Thread GitBox

merlimat commented on issue #2166: 'java.lang.NoSuchMethodError' when running 
standalone
URL: 
https://github.com/apache/incubator-pulsar/issues/2166#issuecomment-405428431
 
 
   @djKooks one other thing that might help is to do `mvn clean` and try again. 
I've seen sometimes a double `mvn install` will leave the target directories in 
a weird state.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] srkukarni commented on issue #2166: 'java.lang.NoSuchMethodError' when running standalone

2018-07-16 Thread GitBox

srkukarni commented on issue #2166: 'java.lang.NoSuchMethodError' when running 
standalone
URL: 
https://github.com/apache/incubator-pulsar/issues/2166#issuecomment-405426077
 
 
   Mac is supported and I tried on mac as well.
   Can you see if you can wipe your ~/.m2/repository and then try again?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] djKooks commented on issue #2166: 'java.lang.NoSuchMethodError' when running standalone

2018-07-16 Thread GitBox

djKooks commented on issue #2166: 'java.lang.NoSuchMethodError' when running 
standalone
URL: 
https://github.com/apache/incubator-pulsar/issues/2166#issuecomment-405424753
 
 
   @srkukarni It is MacOS(with Java8). Is MacOS not supported?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #2013: Unable to consume messages from a partition

2018-07-16 Thread GitBox

sijie commented on issue #2013: Unable to consume messages from a partition
URL: 
https://github.com/apache/incubator-pulsar/issues/2013#issuecomment-405422840
 
 
   @bardock it seems difficult to debug this issue without any logs. it is a 
bit hard to reproduce the behavior.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #2037: Initialization of cluster meta data fails while setting up a multi cluster pulsar instance

2018-07-16 Thread GitBox

sijie commented on issue #2037: Initialization of cluster meta data fails while 
setting up a multi cluster pulsar instance 
URL: 
https://github.com/apache/incubator-pulsar/issues/2037#issuecomment-405422587
 
 
   It seems no updates here. I will close this for now. If there are more 
instructions on how to reproduce it,  open an issue for that, we will 
investigate the issue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie closed issue #2037: Initialization of cluster meta data fails while setting up a multi cluster pulsar instance

2018-07-16 Thread GitBox

sijie closed issue #2037: Initialization of cluster meta data fails while 
setting up a multi cluster pulsar instance 
URL: https://github.com/apache/incubator-pulsar/issues/2037
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #2172: If auth is badly configured in c++, close the connection

2018-07-16 Thread GitBox

merlimat commented on issue #2172: If auth is badly configured in c++, close 
the connection
URL: https://github.com/apache/incubator-pulsar/pull/2172#issuecomment-405414020
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie opened a new pull request #2176: Issue 2119: TopicPatternSubscription doesn't work through proxy

2018-07-16 Thread GitBox

sijie opened a new pull request #2176: Issue 2119: TopicPatternSubscription 
doesn't work through proxy
URL: https://github.com/apache/incubator-pulsar/pull/2176
 
 
   ### Motivation
   
   Fixes #2119 
   
   TopicPatternSubscription is not supported on proxy.
   
   ### Modifications
   
   - add GetTopicsOfNamespace support at the proxy
   - add a unit test to ProxyTest to cover regex subscription
   
   ### Result
   
   Regex subscription is supported at the proxy.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie closed issue #2142: Can't find code source in package org.apache.bookkeeper.mledger.proto

2018-07-16 Thread GitBox

sijie closed issue #2142: Can't find code source in package 
org.apache.bookkeeper.mledger.proto
URL: https://github.com/apache/incubator-pulsar/issues/2142
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #2174: V2 doc changes

2018-07-16 Thread GitBox

merlimat commented on issue #2174: V2 doc changes
URL: https://github.com/apache/incubator-pulsar/pull/2174#issuecomment-405412792
 
 
   > @merlimat Just making sure that I was understanding how this happened 
correctly. Now that I know it was that proposal, I can do another pass through 
the docs for anything using the old scheme. 
   
   Nice, thanks!
   
   > Also, should that proposal be marked "Implemented"?
   
   Yes, we also had plans to move the proposals to git repo (instead of wiki) 
to make it easier for non-committers to submit proposals and comments.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aahmed-se commented on issue #2101: Migrate compaction and s3 offload test to testcontainers

2018-07-16 Thread GitBox

aahmed-se commented on issue #2101: Migrate compaction and s3 offload test to 
testcontainers
URL: https://github.com/apache/incubator-pulsar/pull/2101#issuecomment-405406467
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly opened a new pull request #2175: [WIP] figuring out what's happening to the sha in CI

2018-07-16 Thread GitBox

ivankelly opened a new pull request #2175: [WIP] figuring out what's happening 
to the sha in CI
URL: https://github.com/apache/incubator-pulsar/pull/2175
 
 
   ### Motivation
   
   Explain here the context, and why you're making that change.
   What is the problem you're trying to solve.
   
   ### Modifications
   
   Describe the modifications you've done.
   
   ### Result
   
   After your change, what will change.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] grantwwu commented on issue #2174: V2 doc changes

2018-07-16 Thread GitBox

grantwwu commented on issue #2174: V2 doc changes
URL: https://github.com/apache/incubator-pulsar/pull/2174#issuecomment-405400116
 
 
   @merlimat Just making sure that I was understanding how this happened 
correctly.  Now that I know it was that proposal, I can do another pass through 
the docs for anything using the old scheme.
   
   Also, should that proposal be marked "Implemented"?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #2173: Removed GPL deps on gnu crypto from aerospike connector

2018-07-16 Thread GitBox

merlimat commented on issue #2173: Removed GPL deps on gnu crypto from 
aerospike connector
URL: https://github.com/apache/incubator-pulsar/pull/2173#issuecomment-405399213
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat edited a comment on issue #2174: V2 doc changes

2018-07-16 Thread GitBox

merlimat edited a comment on issue #2174: V2 doc changes
URL: https://github.com/apache/incubator-pulsar/pull/2174#issuecomment-405396654
 
 
   @grantwwu Mostly, to implement that proposal, we had to differentiate the 
API in `/admin/..` and `/admin/v2/..` to avoid naming conflicts. We tried to 
update all the docs, but as you found, we missed some places.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on issue #2174: V2 doc changes

2018-07-16 Thread GitBox

merlimat commented on issue #2174: V2 doc changes
URL: https://github.com/apache/incubator-pulsar/pull/2174#issuecomment-405396654
 
 
   @grantwwu Mostly, to implement that proposal, we had to differentiate the 
API in `/admin/..` and `/admin/v2/..` to avoid naming conflicts.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] grantwwu commented on issue #2174: V2 doc changes

2018-07-16 Thread GitBox

grantwwu commented on issue #2174: V2 doc changes
URL: https://github.com/apache/incubator-pulsar/pull/2174#issuecomment-405395717
 
 
   @merlimat Are all these problems with the cluster in them due to 
https://github.com/apache/incubator-pulsar/wiki/PIP-10:-Remove-cluster-for-namespace-and-topic-names
 ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #2174: V2 doc changes

2018-07-16 Thread GitBox

merlimat commented on a change in pull request #2174: V2 doc changes
URL: https://github.com/apache/incubator-pulsar/pull/2174#discussion_r202833495
 
 

 ##
 File path: site/docs/latest/admin-api/namespaces.md
 ##
 @@ -344,7 +344,7 @@ N/A
 ## REST
 
 ```
-POST /admin/persistent/{tenant}/{namespace}/persistence
+POST /admin/v2/persistent/{tenant}/{namespace}/persistence
 
 Review comment:
   Here the doc was wrong in that the path should be `/admin/v2/namespaces/..` 
instead of `persistent/...`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #2174: V2 doc changes

2018-07-16 Thread GitBox

merlimat commented on a change in pull request #2174: V2 doc changes
URL: https://github.com/apache/incubator-pulsar/pull/2174#discussion_r202832925
 
 

 ##
 File path: site/docs/latest/admin-api/namespaces.md
 ##
 @@ -47,7 +47,7 @@ $ pulsar-admin namespaces create test-tenant/test-namespace
 
  REST API
 
-{% endpoint PUT /admin/namespaces/:tenant/:cluster/:namespace %}
+{% endpoint PUT /admin/v2/namespaces/:tenant/:cluster/:namespace %}
 
 Review comment:
   `/admin/v2/namespaces/:tenant/:namespace`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #2174: V2 doc changes

2018-07-16 Thread GitBox

merlimat commented on a change in pull request #2174: V2 doc changes
URL: https://github.com/apache/incubator-pulsar/pull/2174#discussion_r202833084
 
 

 ##
 File path: site/docs/latest/admin-api/namespaces.md
 ##
 @@ -144,7 +144,7 @@ test-tenant/ns2
 
  REST API
 
-{% endpoint GET /admin/namespaces/:tenant/:cluster %}
+{% endpoint GET /admin/v2/namespaces/:tenant/:cluster %}
 
 Review comment:
   This handle was removed in v2 API, so we should remove from here


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #2174: V2 doc changes

2018-07-16 Thread GitBox

merlimat commented on a change in pull request #2174: V2 doc changes
URL: https://github.com/apache/incubator-pulsar/pull/2174#discussion_r202833153
 
 

 ##
 File path: site/docs/latest/admin-api/namespaces.md
 ##
 @@ -168,7 +168,7 @@ $ pulsar-admin namespaces delete test-tenant/ns1
 
  REST
 
-{% endpoint DELETE /admin/namespaces/:tenant/:cluster/:namespace %}
+{% endpoint DELETE /admin/v2/namespaces/:tenant/:cluster/:namespace %}
 
 Review comment:
   `/admin/v2/namespaces/:cluster/:namespace`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #2174: V2 doc changes

2018-07-16 Thread GitBox

merlimat commented on a change in pull request #2174: V2 doc changes
URL: https://github.com/apache/incubator-pulsar/pull/2174#discussion_r202833182
 
 

 ##
 File path: site/docs/latest/admin-api/namespaces.md
 ##
 @@ -92,7 +92,7 @@ $ pulsar-admin namespaces policies test-tenant/test-namespace
 
  REST API
 
-{% endpoint GET /admin/namespaces/:tenant/:cluster/:namespace %}
+{% endpoint GET /admin/v2/namespaces/:tenant/:cluster/:namespace %}
 
 Review comment:
   /admin/v2/namespaces/:tenant/:namespace


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #2174: V2 doc changes

2018-07-16 Thread GitBox

merlimat commented on a change in pull request #2174: V2 doc changes
URL: https://github.com/apache/incubator-pulsar/pull/2174#discussion_r202831818
 
 

 ##
 File path: site/docs/latest/cookbooks/RetentionExpiry.md
 ##
 @@ -97,7 +97,7 @@ $ pulsar-admin namespaces set-retention 
my-prop/my-cluster/my-ns \
 
  REST API
 
-{% endpoint POST /admin/namespaces/:property/:cluster/:namespace/retention %}
+{% endpoint POST /admin/v2/namespaces/:property/:cluster/:namespace/retention 
%}
 
 Review comment:
   And the terminology was changed from `property` into `tenant`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat commented on a change in pull request #2174: V2 doc changes

2018-07-16 Thread GitBox

merlimat commented on a change in pull request #2174: V2 doc changes
URL: https://github.com/apache/incubator-pulsar/pull/2174#discussion_r202831664
 
 

 ##
 File path: site/docs/latest/cookbooks/RetentionExpiry.md
 ##
 @@ -97,7 +97,7 @@ $ pulsar-admin namespaces set-retention 
my-prop/my-cluster/my-ns \
 
  REST API
 
-{% endpoint POST /admin/namespaces/:property/:cluster/:namespace/retention %}
+{% endpoint POST /admin/v2/namespaces/:property/:cluster/:namespace/retention 
%}
 
 Review comment:
   In all v2 APIs, the portion `:property/:cluster/:namespace` would actually 
be `:property/:namespace`, since we actually took the cluster out of the 
namespaces (and topic) names.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] grantwwu commented on issue #2174: V2 doc changes

2018-07-16 Thread GitBox

grantwwu commented on issue #2174: V2 doc changes
URL: https://github.com/apache/incubator-pulsar/pull/2174#issuecomment-405382916
 
 
   @merlimat @sijie Sorry I just added on a bunch more and squashed into a 
single commit.
   
   Should we mention anywhere that there's still a v1 API?  Perhaps link to the 
latest 1.x release?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: Add kinesis-sink user metrics to sinkContext (#2169)

2018-07-16 Thread rdhabalia

This is an automated email from the ASF dual-hosted git repository.

rdhabalia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new ba1ea66  Add kinesis-sink user metrics to sinkContext (#2169)
ba1ea66 is described below

commit ba1ea665f0a073701df68eeff3a6b2541f6d1fa6
Author: Rajan Dhabalia 
AuthorDate: Mon Jul 16 14:00:46 2018 -0700

Add kinesis-sink user metrics to sinkContext (#2169)
---
 .../org/apache/pulsar/io/kinesis/KinesisSink.java  | 34 ++
 1 file changed, 28 insertions(+), 6 deletions(-)

diff --git 
a/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java 
b/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java
index c3b6cdc..c3c2c45 100644
--- 
a/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java
+++ 
b/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java
@@ -88,21 +88,32 @@ public class KinesisSink implements Sink {
 private String streamName;
 private static final String defaultPartitionedKey = "default";
 private static final int maxPartitionedKeyLength = 256;
+private SinkContext sinkContext;
 
 public static final String ACCESS_KEY_NAME = "accessKey";
 public static final String SECRET_KEY_NAME = "secretKey";
 
+public static final String METRICS_TOTAL_INCOMING = 
"_kinesis_total_incoming_";
+public static final String METRICS_TOTAL_INCOMING_BYTES = 
"_kinesis_total_incoming_bytes_";
+public static final String METRICS_TOTAL_SUCCESS = 
"_kinesis_total_success_";
+public static final String METRICS_TOTAL_FAILURE = 
"_kinesis_total_failure_";
+ 
+
 @Override
 public void write(Record record) throws Exception {
 String partitionedKey = record.getKey().orElse(defaultPartitionedKey);
 partitionedKey = partitionedKey.length() > maxPartitionedKeyLength
 ? partitionedKey.substring(0, maxPartitionedKeyLength - 1)
 : partitionedKey; // partitionedKey Length must be at least 
one, and at most 256
+ByteBuffer data = 
createKinesisMessage(kinesisSinkConfig.getMessageFormat(), record);
 ListenableFuture addRecordResult = 
kinesisProducer.addUserRecord(this.streamName,
-partitionedKey,
-createKinesisMessage(kinesisSinkConfig.getMessageFormat(), 
record));
+partitionedKey, data);
 addCallback(addRecordResult,
-ProducerSendCallback.create(this.streamName, record, 
System.nanoTime()), directExecutor());
+ProducerSendCallback.create(this.streamName, record, 
System.nanoTime(), sinkContext), directExecutor());
+if (sinkContext != null) {
+sinkContext.recordMetric(METRICS_TOTAL_INCOMING, 1);
+sinkContext.recordMetric(METRICS_TOTAL_INCOMING_BYTES, 
data.array().length);
+}
 if (LOG.isDebugEnabled()) {
 LOG.debug("Published message to kinesis stream {} with size {}", 
streamName, record.getValue().length);
 }
@@ -120,6 +131,7 @@ public class KinesisSink implements Sink {
 @Override
 public void open(Map config, SinkContext sinkContext) 
throws Exception {
 kinesisSinkConfig = KinesisSinkConfig.load(config);
+this.sinkContext = sinkContext;
 
 checkArgument(isNotBlank(kinesisSinkConfig.getAwsKinesisStreamName()), 
"empty kinesis-stream name");
 checkArgument(isNotBlank(kinesisSinkConfig.getAwsEndpoint()), "empty 
aws-end-point");
@@ -158,16 +170,19 @@ public class KinesisSink implements Sink {
 private String streamName;
 private long startTime = 0;
 private final Handle recyclerHandle;
+private SinkContext sinkContext;
 
 private ProducerSendCallback(Handle 
recyclerHandle) {
 this.recyclerHandle = recyclerHandle;
 }
 
-static ProducerSendCallback create(String streamName, Record 
resultContext, long startTime) {
+static ProducerSendCallback create(String streamName, Record 
resultContext, long startTime,
+SinkContext sinkContext) {
 ProducerSendCallback sendCallback = RECYCLER.get();
 sendCallback.resultContext = resultContext;
 sendCallback.streamName = streamName;
 sendCallback.startTime = startTime;
+sendCallback.sinkContext = sinkContext;
 return sendCallback;
 }
 
@@ -175,6 +190,7 @@ public class KinesisSink implements Sink {
 resultContext = null;
 streamName = null;
 startTime = 0;
+sinkContext = null;
 recyclerHandle.recycle(this);
 }
 
@@ -188,10 +204,13 @@ public class KinesisSink implements Sink {
 @Override
 public void onSuccess(UserRecordResult result) {
 if (LOG.isDeb

[GitHub] rdhabalia closed pull request #2169: Add kinesis-sink user metrics to sinkContext

2018-07-16 Thread GitBox

rdhabalia closed pull request #2169: Add kinesis-sink user metrics to 
sinkContext
URL: https://github.com/apache/incubator-pulsar/pull/2169
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java 
b/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java
index c3b6cdcd26..c3c2c452fb 100644
--- 
a/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java
+++ 
b/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisSink.java
@@ -88,21 +88,32 @@
 private String streamName;
 private static final String defaultPartitionedKey = "default";
 private static final int maxPartitionedKeyLength = 256;
+private SinkContext sinkContext;
 
 public static final String ACCESS_KEY_NAME = "accessKey";
 public static final String SECRET_KEY_NAME = "secretKey";
 
+public static final String METRICS_TOTAL_INCOMING = 
"_kinesis_total_incoming_";
+public static final String METRICS_TOTAL_INCOMING_BYTES = 
"_kinesis_total_incoming_bytes_";
+public static final String METRICS_TOTAL_SUCCESS = 
"_kinesis_total_success_";
+public static final String METRICS_TOTAL_FAILURE = 
"_kinesis_total_failure_";
+ 
+
 @Override
 public void write(Record record) throws Exception {
 String partitionedKey = record.getKey().orElse(defaultPartitionedKey);
 partitionedKey = partitionedKey.length() > maxPartitionedKeyLength
 ? partitionedKey.substring(0, maxPartitionedKeyLength - 1)
 : partitionedKey; // partitionedKey Length must be at least 
one, and at most 256
+ByteBuffer data = 
createKinesisMessage(kinesisSinkConfig.getMessageFormat(), record);
 ListenableFuture addRecordResult = 
kinesisProducer.addUserRecord(this.streamName,
-partitionedKey,
-createKinesisMessage(kinesisSinkConfig.getMessageFormat(), 
record));
+partitionedKey, data);
 addCallback(addRecordResult,
-ProducerSendCallback.create(this.streamName, record, 
System.nanoTime()), directExecutor());
+ProducerSendCallback.create(this.streamName, record, 
System.nanoTime(), sinkContext), directExecutor());
+if (sinkContext != null) {
+sinkContext.recordMetric(METRICS_TOTAL_INCOMING, 1);
+sinkContext.recordMetric(METRICS_TOTAL_INCOMING_BYTES, 
data.array().length);
+}
 if (LOG.isDebugEnabled()) {
 LOG.debug("Published message to kinesis stream {} with size {}", 
streamName, record.getValue().length);
 }
@@ -120,6 +131,7 @@ public void close() throws IOException {
 @Override
 public void open(Map config, SinkContext sinkContext) 
throws Exception {
 kinesisSinkConfig = KinesisSinkConfig.load(config);
+this.sinkContext = sinkContext;
 
 checkArgument(isNotBlank(kinesisSinkConfig.getAwsKinesisStreamName()), 
"empty kinesis-stream name");
 checkArgument(isNotBlank(kinesisSinkConfig.getAwsEndpoint()), "empty 
aws-end-point");
@@ -158,16 +170,19 @@ protected AWSCredentialsProvider 
createCredentialProvider(String awsCredentialPl
 private String streamName;
 private long startTime = 0;
 private final Handle recyclerHandle;
+private SinkContext sinkContext;
 
 private ProducerSendCallback(Handle 
recyclerHandle) {
 this.recyclerHandle = recyclerHandle;
 }
 
-static ProducerSendCallback create(String streamName, Record 
resultContext, long startTime) {
+static ProducerSendCallback create(String streamName, Record 
resultContext, long startTime,
+SinkContext sinkContext) {
 ProducerSendCallback sendCallback = RECYCLER.get();
 sendCallback.resultContext = resultContext;
 sendCallback.streamName = streamName;
 sendCallback.startTime = startTime;
+sendCallback.sinkContext = sinkContext;
 return sendCallback;
 }
 
@@ -175,6 +190,7 @@ private void recycle() {
 resultContext = null;
 streamName = null;
 startTime = 0;
+sinkContext = null;
 recyclerHandle.recycle(this);
 }
 
@@ -188,10 +204,13 @@ protected ProducerSendCallback 
newObject(Handle handle) {
 @Override
 public void onSuccess(UserRecordResult result) {
 if (LOG.isDebugEnabled()) {
-LOG.debug("Successfully published message for replicator of 
{}-{} with latency", this.streamName,
-result.getShardId(), 
TimeUnit.NANOSECONDS.toMillis((System.nanoTime() - startTime)));
+

[incubator-pulsar] branch master updated: Add function metrics with function-stats to get metrics on-demand (#2130)

2018-07-16 Thread rdhabalia

This is an automated email from the ASF dual-hosted git repository.

rdhabalia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 5f779b4  Add function metrics with function-stats to get metrics 
on-demand (#2130)
5f779b4 is described below

commit 5f779b4ce541e1201354ca5454bebf1276e93f0b
Author: Rajan Dhabalia 
AuthorDate: Mon Jul 16 14:00:07 2018 -0700

Add function metrics with function-stats to get metrics on-demand (#2130)
---
 .../org/apache/pulsar/io/PulsarSinkE2ETest.java| 84 --
 .../functions/instance/JavaInstanceRunnable.java   | 22 --
 .../src/main/python/InstanceCommunication_pb2.py   | 44 +++-
 .../instance/src/main/python/python_instance.py|  1 +
 .../src/main/proto/InstanceCommunication.proto |  1 +
 .../pulsar/functions/runtime/ThreadRuntime.java| 13 +++-
 6 files changed, 133 insertions(+), 32 deletions(-)

diff --git 
a/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java 
b/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java
index 5d70525..3ece269 100644
--- a/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java
+++ b/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java
@@ -48,22 +48,26 @@ import org.apache.pulsar.client.api.ClientBuilder;
 import org.apache.pulsar.client.api.Consumer;
 import org.apache.pulsar.client.api.Message;
 import org.apache.pulsar.client.api.Producer;
-import org.apache.pulsar.client.api.ProducerBuilder;
 import org.apache.pulsar.client.api.PulsarClient;
-import org.apache.pulsar.client.api.TypedMessageBuilder;
 import org.apache.pulsar.client.impl.auth.AuthenticationTls;
 import org.apache.pulsar.common.policies.data.ClusterData;
 import org.apache.pulsar.common.policies.data.SubscriptionStats;
 import org.apache.pulsar.common.policies.data.TenantInfo;
 import org.apache.pulsar.functions.api.utils.DefaultSerDe;
 import org.apache.pulsar.functions.api.utils.IdentityFunction;
+import org.apache.pulsar.functions.instance.JavaInstanceRunnable;
 import org.apache.pulsar.functions.proto.Function;
 import org.apache.pulsar.functions.proto.Function.FunctionDetails;
 import org.apache.pulsar.functions.proto.Function.SinkSpec;
 import org.apache.pulsar.functions.proto.Function.SourceSpec;
+import org.apache.pulsar.functions.proto.InstanceCommunication.FunctionStatus;
+import 
org.apache.pulsar.functions.proto.InstanceCommunication.FunctionStatusList;
+import org.apache.pulsar.functions.proto.InstanceCommunication.MetricsData;
+import 
org.apache.pulsar.functions.proto.InstanceCommunication.MetricsData.DataDigest;
 import org.apache.pulsar.functions.sink.PulsarSink;
 import org.apache.pulsar.functions.utils.Reflections;
 import org.apache.pulsar.functions.utils.Utils;
+import org.apache.pulsar.functions.worker.FunctionRuntimeManager;
 import org.apache.pulsar.functions.worker.WorkerConfig;
 import org.apache.pulsar.functions.worker.WorkerService;
 import org.apache.pulsar.functions.worker.rest.WorkerServer;
@@ -244,6 +248,7 @@ public class PulsarSinkE2ETest {
 final String sinkTopic = "persistent://" + replNamespace + "/output";
 final String propertyKey = "key";
 final String propertyValue = "value";
+final String functionName = "PulsarSink-test";
 admin.namespaces().createNamespace(replNamespace);
 Set clusters = Sets.newHashSet(Lists.newArrayList("use"));
 admin.namespaces().setNamespaceReplicationClusters(replNamespace, 
clusters);
@@ -254,7 +259,7 @@ public class PulsarSinkE2ETest {
 
 String jarFilePathUrl = Utils.FILE + ":"
 + 
PulsarSink.class.getProtectionDomain().getCodeSource().getLocation().getPath();
-FunctionDetails functionDetails = createSinkConfig(jarFilePathUrl, 
tenant, namespacePortion, "PulsarSink-test",
+FunctionDetails functionDetails = createSinkConfig(jarFilePathUrl, 
tenant, namespacePortion, functionName,
 sinkTopic);
 admin.functions().createFunctionWithUrl(functionDetails, 
jarFilePathUrl);
 
@@ -298,7 +303,76 @@ public class PulsarSinkE2ETest {
 
 }
 
-protected FunctionDetails createSinkConfig(String jarFile, String tenant, 
String namespace, String sinkName, String sinkTopic) {
+
+@Test(timeOut = 2)
+public void testPulsarSinkStats() throws Exception {
+
+final String namespacePortion = "io";
+final String replNamespace = tenant + "/" + namespacePortion;
+final String sourceTopic = "persistent://" + replNamespace + 
"/my-topic1";
+final String sinkTopic = "persistent://" + replNamespace + "/output";
+final String propertyKey = "key";
+final String propertyValue = "value";
+final String functionName = "PulsarSink-test";
+admin.namespaces().createNamespace(replNamespace);
+

[GitHub] sijie commented on a change in pull request #2101: Migrate compaction and s3 offload test to testcontainers

2018-07-16 Thread GitBox

sijie commented on a change in pull request #2101: Migrate compaction and s3 
offload test to testcontainers
URL: https://github.com/apache/incubator-pulsar/pull/2101#discussion_r202815746
 
 

 ##
 File path: 
tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/topologies/PulsarClusterSpec.java
 ##
 @@ -90,13 +90,14 @@
  *
  * @return the list of external services to start with the cluster.
  */
-Map> externalServices = Maps.newHashMap();
+Map> externalServices = Collections.EMPTY_MAP;
 
 Review comment:
   sure. I might be using the wrong class. however the point is don't mixing 
fixes into one PR. if this is wrong, it is ideally to be fixed in a separate PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #2101: Migrate compaction and s3 offload test to testcontainers

2018-07-16 Thread GitBox

sijie commented on issue #2101: Migrate compaction and s3 offload test to 
testcontainers
URL: https://github.com/apache/incubator-pulsar/pull/2101#issuecomment-405374084
 
 
   @ivankelly can you review this again?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #2170: Add integration test for kafka sink

2018-07-16 Thread GitBox

sijie commented on issue #2170: Add integration test for kafka sink
URL: https://github.com/apache/incubator-pulsar/pull/2170#issuecomment-405370838
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aahmed-se closed pull request #2036: Add testcontainer based standalone test

2018-07-16 Thread GitBox

aahmed-se closed pull request #2036: Add testcontainer based standalone test
URL: https://github.com/apache/incubator-pulsar/pull/2036
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/tests/integration-tc/pom.xml b/tests/integration-tc/pom.xml
new file mode 100644
index 00..45e7487a0f
--- /dev/null
+++ b/tests/integration-tc/pom.xml
@@ -0,0 +1,140 @@
+
+
+http://maven.apache.org/POM/4.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="
+  http://maven.apache.org/POM/4.0.0
+  http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+4.0.0
+
+org.apache.pulsar.tests
+tests-parent
+2.2.0-incubating-SNAPSHOT
+
+
+org.apache.pulsar.tests
+integration-tc
+jar
+
+Apache Pulsar :: Tests :: Container Based integration tests
+
+
+1.8.0
+1.15
+
+
+
+
+org.apache.commons
+commons-compress
+${commons-compress.version}
+
+
+
+org.apache.zookeeper
+zookeeper
+
+
+
+org.slf4j
+slf4j-api
+
+
+
+org.apache.logging.log4j
+log4j-slf4j-impl
+
+
+
+com.fasterxml.jackson.core
+jackson-databind
+
+
+
+com.fasterxml.jackson.dataformat
+jackson-dataformat-yaml
+
+
+
+org.testcontainers
+testcontainers
+1.8.0
+
+
+
+org.apache.pulsar
+pulsar-client
+${project.version}
+
+
+
+
+
+
+
+org.apache.maven.plugins
+maven-surefire-plugin
+
+
+true
+
+${project.version}
+
${project.build.directory}
+
+
+
+
+
+
+
+
+integrationTests
+
+
+integrationTests
+
+
+
+
+
+org.apache.maven.plugins
+maven-surefire-plugin
+
+
+
+listener
+
+
org.apache.pulsar.tests.PulsarTestListener
+
+
+
+-Xmx2G -XX:MaxDirectMemorySize=8G
+-Dio.netty.leakDetectionLevel=advanced
+
+false
+
+
+
+
+
+
+
diff --git 
a/tests/integration-tc/src/main/java/org/apache/pulsar/tests/integration/FUNC_MODE.java
 
b/tests/integration-tc/src/main/java/org/apache/pulsar/tests/integration/FUNC_MODE.java
new file mode 100644
index 00..c23a7f42dd
--- /dev/null
+++ 
b/tests/integration-tc/src/main/java/org/apache/pulsar/tests/integration/FUNC_MODE.java
@@ -0,0 +1,6 @@
+package org.apache.pulsar.tests.integration;
+
+public enum FUNC_MODE {
+BROKER_CLUSTER,
+STANDALONE;
+}
\ No newline at end of file
diff --git 
a/tests/integration-tc/src/main/java/org/apache/pulsar/tests/integration/PulsarStandAloneCluster.java
 
b/tests/integration-tc/src/main/java/org/apache/pulsar/tests/integration/PulsarStandAloneCluster.java
new file mode 100644
index 00..07d9de93d4
--- /dev/null
+++ 
b/tests/integration-tc/src/main/java/org/apache/pulsar/tests/integration/PulsarStandAloneCluster.java
@@ -0,0 +1,121 @@
+package org.apache.pulsar.tests.integration;
+
+import com.github.dockerjava.api.command.CreateContainerCmd;
+import com.github.dockerjava.api.command.LogContainerCmd;
+import com.github.dockerjava.api.model.Frame;
+import com.github.dockerjava.core.command.LogContainerResultCallback;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.output.Slf4jLogConsumer;
+import org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Consumer;
+
+import static java.time.temporal.ChronoUnit.SECONDS;
+
+public class PulsarStandAloneCluster extends 
GenericContainer {
+
+private static final Logger LOG = 
LoggerFactory.getLogger(PulsarStandAloneCluster.class);
+
+private static

[GitHub] aahmed-se commented on issue #2036: Add testcontainer based standalone test

2018-07-16 Thread GitBox

aahmed-se commented on issue #2036: Add testcontainer based standalone test
URL: https://github.com/apache/incubator-pulsar/pull/2036#issuecomment-405365008
 
 
   This is out of context now will close it for now.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] grantwwu opened a new pull request #2174: V2 doc changes

2018-07-16 Thread GitBox

grantwwu opened a new pull request #2174: V2 doc changes
URL: https://github.com/apache/incubator-pulsar/pull/2174
 
 
   Apparently we've forgotten to add some v2s to the URLs.
   
   NOTE: I think there will be more changes needed.   Also - should we note the 
availability of v1 of the API somewhere?
   
   ### Motivation
   
   Docs are missing critical v2 URL component for APIs!
   
   ### Modifications
   
   Add v2 to a bunch of URLs.
   
   ### Result
   
   Docs will be more accurate.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aahmed-se commented on issue #2101: Migrate compaction and s3 offload test to testcontainers

2018-07-16 Thread GitBox

aahmed-se commented on issue #2101: Migrate compaction and s3 offload test to 
testcontainers
URL: https://github.com/apache/incubator-pulsar/pull/2101#issuecomment-405358779
 
 
   Conflicts should be fixed now


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aahmed-se commented on a change in pull request #2101: Migrate compaction and s3 offload test to testcontainers

2018-07-16 Thread GitBox

aahmed-se commented on a change in pull request #2101: Migrate compaction and 
s3 offload test to testcontainers
URL: https://github.com/apache/incubator-pulsar/pull/2101#discussion_r202799087
 
 

 ##
 File path: 
tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/topologies/PulsarCluster.java
 ##
 @@ -193,24 +208,25 @@ public void start() throws Exception {
 String name = "pulsar-" + serviceName + "-" + i;
 T container = containerCreator.apply(name);
 containers.put(name, container);
-startFutures.add(CompletableFuture.runAsync(() -> 
container.start()));
 }
-CompletableFuture.allOf(startFutures.toArray(new 
CompletableFuture[startFutures.size()])).join();
-log.info("Successfully started {} {} containers", numContainers, 
serviceName);
 return containers;
 }
 
 public void stop() {
-Stream.of(proxyContainer, csContainer, 
zkContainer).parallel().forEach(GenericContainer::stop);
-
workerContainers.values().parallelStream().forEach(GenericContainer::stop);
-
brokerContainers.values().parallelStream().forEach(GenericContainer::stop);
-
bookieContainers.values().parallelStream().forEach(GenericContainer::stop);
-if (null != spec.externalServices()) {
-spec.externalServices().values()
-.parallelStream()
-.forEach(GenericContainer::stop);
+
+Stream containers = Streams.concat(
 
 Review comment:
   it's meant to parallelize all container shutdown.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aahmed-se commented on a change in pull request #2101: Migrate compaction and s3 offload test to testcontainers

2018-07-16 Thread GitBox

aahmed-se commented on a change in pull request #2101: Migrate compaction and 
s3 offload test to testcontainers
URL: https://github.com/apache/incubator-pulsar/pull/2101#discussion_r202798871
 
 

 ##
 File path: 
tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/topologies/PulsarCluster.java
 ##
 @@ -176,15 +189,17 @@ public void start() throws Exception {
 
 // start external services
 Map> externalServices = 
spec.externalServices;
-if (null != externalServices) {
-externalServices.entrySet().forEach(service -> {
+if (externalServices != null) {
 
 Review comment:
   we are adding withnetwork and withNetworkAliases to make external services 
work.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aahmed-se commented on a change in pull request #2101: Migrate compaction and s3 offload test to testcontainers

2018-07-16 Thread GitBox

aahmed-se commented on a change in pull request #2101: Migrate compaction and 
s3 offload test to testcontainers
URL: https://github.com/apache/incubator-pulsar/pull/2101#discussion_r202797674
 
 

 ##
 File path: 
tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/topologies/PulsarClusterSpec.java
 ##
 @@ -90,13 +90,14 @@
  *
  * @return the list of external services to start with the cluster.
  */
-Map> externalServices = Maps.newHashMap();
+Map> externalServices = Collections.EMPTY_MAP;
 
 Review comment:
   you are taking a dependency from testng in non testcode, that's not correct.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on a change in pull request #2065: GCS offload support(2): replace `s3client` api with `jclouds` related api

2018-07-16 Thread GitBox

ivankelly commented on a change in pull request #2065: GCS offload support(2): 
replace `s3client` api with `jclouds` related api 
URL: https://github.com/apache/incubator-pulsar/pull/2065#discussion_r202795291
 
 

 ##
 File path: 
pulsar-broker/src/main/java/org/apache/pulsar/broker/offload/impl/BlobStoreBackedInputStreamImpl.java
 ##
 @@ -76,26 +73,27 @@ private boolean refillBufferIfNeeded() throws IOException {
 long startRange = cursor;
 long endRange = Math.min(cursor + bufferSize - 1,
  objectLen - 1);
-GetObjectRequest req = new GetObjectRequest(bucket, key)
-.withRange(startRange, endRange);
-log.debug("Reading range {}-{} from {}/{}", startRange, endRange, 
bucket, key);
-try (S3Object obj = s3client.getObject(req)) {
-versionCheck.check(key, obj.getObjectMetadata());
-
-Long[] range = obj.getObjectMetadata().getContentRange();
-long bytesRead = range[1] - range[0] + 1;
 
-buffer.clear();
-bufferOffsetStart = range[0];
-bufferOffsetEnd = range[1];
-InputStream s = obj.getObjectContent();
-int bytesToCopy = (int)bytesRead;
-while (bytesToCopy > 0) {
-bytesToCopy -= buffer.writeBytes(s, bytesToCopy);
+try {
+Blob blob = blobStore.getBlob(bucket, key);
+versionCheck.check(key, blob);
+PayloadSlicer slicer = new BasePayloadSlicer();
 
 Review comment:
   Slicer isnt what you need here. getBlob can take a GetOptions parameter. You 
can set the range in that.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rdhabalia removed a comment on issue #2169: Add kinesis-sink user metrics to sinkContext

2018-07-16 Thread GitBox

rdhabalia removed a comment on issue #2169: Add kinesis-sink user metrics to 
sinkContext
URL: https://github.com/apache/incubator-pulsar/pull/2169#issuecomment-405352052
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] rdhabalia commented on issue #2169: Add kinesis-sink user metrics to sinkContext

2018-07-16 Thread GitBox

rdhabalia commented on issue #2169: Add kinesis-sink user metrics to sinkContext
URL: https://github.com/apache/incubator-pulsar/pull/2169#issuecomment-405352052
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] merlimat opened a new pull request #2173: Removed GPL deps on gnu crypto from aerospike connector

2018-07-16 Thread GitBox

merlimat opened a new pull request #2173: Removed GPL deps on gnu crypto from 
aerospike connector
URL: https://github.com/apache/incubator-pulsar/pull/2173
 
 
   ### Motivation
   
   There's an optional dependency on gnu-crypto for aerospike connector. We 
need to remove it to avoid license issue at release.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #2172: If auth is badly configured in c++, close the connection

2018-07-16 Thread GitBox

sijie commented on issue #2172: If auth is badly configured in c++, close the 
connection
URL: https://github.com/apache/incubator-pulsar/pull/2172#issuecomment-405345234
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #2170: Add integration test for kafka sink

2018-07-16 Thread GitBox

sijie commented on issue #2170: Add integration test for kafka sink
URL: https://github.com/apache/incubator-pulsar/pull/2170#issuecomment-405344833
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #2171: Add integration test for kafka source

2018-07-16 Thread GitBox

sijie commented on issue #2171: Add integration test for kafka source
URL: https://github.com/apache/incubator-pulsar/pull/2171#issuecomment-405344972
 
 
   @merlimat #2170 need to go first. can you review that?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aahmed-se commented on a change in pull request #2101: Migrate compaction and s3 offload test to testcontainers

2018-07-16 Thread GitBox

aahmed-se commented on a change in pull request #2101: Migrate compaction and 
s3 offload test to testcontainers
URL: https://github.com/apache/incubator-pulsar/pull/2101#discussion_r202786737
 
 

 ##
 File path: 
tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/topologies/PulsarCluster.java
 ##
 @@ -265,6 +282,13 @@ private void 
startFunctionWorkersWithThreadContainerFactory(int numFunctionWorke
 // bookkeeper tools
 .withEnv("zkServers", ZKContainer.NAME)
 ));
+this.startWorkers();
+}
+
+private void startWorkers() {
+// Start workers that have been initialized
+workerContainers.values().stream().forEach(WorkerContainer::start);
 
 Review comment:
   it's meant to be parallel init


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aahmed-se commented on a change in pull request #2101: Migrate compaction and s3 offload test to testcontainers

2018-07-16 Thread GitBox

aahmed-se commented on a change in pull request #2101: Migrate compaction and 
s3 offload test to testcontainers
URL: https://github.com/apache/incubator-pulsar/pull/2101#discussion_r202786268
 
 

 ##
 File path: 
tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/topologies/PulsarCluster.java
 ##
 @@ -193,24 +208,25 @@ public void start() throws Exception {
 String name = "pulsar-" + serviceName + "-" + i;
 T container = containerCreator.apply(name);
 containers.put(name, container);
-startFutures.add(CompletableFuture.runAsync(() -> 
container.start()));
 }
-CompletableFuture.allOf(startFutures.toArray(new 
CompletableFuture[startFutures.size()])).join();
-log.info("Successfully started {} {} containers", numContainers, 
serviceName);
 return containers;
 }
 
 public void stop() {
-Stream.of(proxyContainer, csContainer, 
zkContainer).parallel().forEach(GenericContainer::stop);
-
workerContainers.values().parallelStream().forEach(GenericContainer::stop);
-
brokerContainers.values().parallelStream().forEach(GenericContainer::stop);
-
bookieContainers.values().parallelStream().forEach(GenericContainer::stop);
-if (null != spec.externalServices()) {
-spec.externalServices().values()
-.parallelStream()
-.forEach(GenericContainer::stop);
+
+Stream containers = Streams.concat(
+workerContainers.values().stream(),
+brokerContainers.values().stream(),
+bookieContainers.values().stream(),
+Stream.of(proxyContainer, csContainer, zkContainer)
+);
+
+if (spec.externalServices() != null) {
+containers = Streams.concat(containers, 
spec.externalServices().values().stream());
 }
 
+containers.forEach(GenericContainer::stop);
 
 Review comment:
   it's parallel now


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aahmed-se commented on a change in pull request #2101: Migrate compaction and s3 offload test to testcontainers

2018-07-16 Thread GitBox

aahmed-se commented on a change in pull request #2101: Migrate compaction and 
s3 offload test to testcontainers
URL: https://github.com/apache/incubator-pulsar/pull/2101#discussion_r202785971
 
 

 ##
 File path: 
tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/topologies/PulsarCluster.java
 ##
 @@ -20,18 +20,17 @@
 
 import static com.google.common.base.Preconditions.checkArgument;
 import static org.apache.pulsar.tests.containers.PulsarContainer.CS_PORT;
+import static org.apache.pulsar.tests.containers.PulsarContainer.ZK_PORT;
 
 import com.google.common.collect.Lists;
 import com.google.common.collect.Maps;
 
-import java.util.Collection;
-import java.util.Collections;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aahmed-se commented on a change in pull request #2101: Migrate compaction and s3 offload test to testcontainers

2018-07-16 Thread GitBox

aahmed-se commented on a change in pull request #2101: Migrate compaction and 
s3 offload test to testcontainers
URL: https://github.com/apache/incubator-pulsar/pull/2101#discussion_r202786021
 
 

 ##
 File path: 
tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/BrokerContainer.java
 ##
 @@ -18,6 +18,10 @@
  */
 package org.apache.pulsar.tests.containers;
 
+import org.apache.pulsar.tests.topologies.PulsarCluster;
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly commented on a change in pull request #2135: Allow user to get topic from consumer message (#2007)

2018-07-16 Thread GitBox

ivankelly commented on a change in pull request #2135: Allow user to get topic 
from consumer message (#2007)
URL: https://github.com/apache/incubator-pulsar/pull/2135#discussion_r202763305
 
 

 ##
 File path: 
pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java
 ##
 @@ -333,6 +333,18 @@ public String getSchemaName() {
 + "/" + getLocalName();
 }
 
+/**
+ * Get the full name for the topic.
+ * E.g. persistent://public/default/topic1
+ */
+public String getFullName() {
+if (isV2()) {
+return String.format("%s://%s/%s/%s", domain, tenant, 
namespacePortion, getEncodedLocalName());
+} else {
+return String.format("%s://%s/%s/%s/%s", domain, tenant, cluster, 
namespacePortion, getEncodedLocalName());
+}
+}
 
 Review comment:
   @merlimat changed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] srkukarni commented on a change in pull request #2105: Attach function metadata properties to pulsar source/sink

2018-07-16 Thread GitBox

srkukarni commented on a change in pull request #2105: Attach function metadata 
properties to pulsar source/sink
URL: https://github.com/apache/incubator-pulsar/pull/2105#discussion_r202754104
 
 

 ##
 File path: 
pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java
 ##
 @@ -194,4 +207,12 @@ void setupSerDe() throws ClassNotFoundException {
 }
 }
 }
+
+public static String getFunctionNameString(FunctionDetails fd) {
+if (fd == null) {
+return "";
+}
+
+return String.format("%s/%s/%s", fd.getTenant(), fd.getNamespace(), 
fd.getName());
 
 Review comment:
   I beleive there is already a utility function to get the fqn


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] srkukarni commented on issue #2166: 'java.lang.NoSuchMethodError' when running standalone

2018-07-16 Thread GitBox

srkukarni commented on issue #2166: 'java.lang.NoSuchMethodError' when running 
standalone
URL: 
https://github.com/apache/incubator-pulsar/issues/2166#issuecomment-405312563
 
 
   I built master and did not encounter any such issue. What's your os/java 
version? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch master updated: fixing protobuf schema shading issue (#2168)

2018-07-16 Thread jerrypeng

This is an automated email from the ASF dual-hosted git repository.

jerrypeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/master by this push:
 new 7af6374   fixing protobuf schema shading issue (#2168)
7af6374 is described below

commit 7af63740c1586cb79925fb8b5884bb8e51097d09
Author: Boyang Jerry Peng 
AuthorDate: Mon Jul 16 09:18:36 2018 -0700

 fixing protobuf schema shading issue (#2168)

* fixing protobuf schema shading issue
---
 pom.xml|   1 +
 pulsar-broker/pom.xml  |   5 +
 pulsar-client-admin-shaded/pom.xml |  30 --
 .../pulsar-client-kafka-shaded/pom.xml |  31 +-
 pulsar-client-schema/pom.xml   | 349 +
 .../java/org/apache/pulsar/client/api/Schema.java  |  16 +
 .../client/api/SchemaSerializationException.java   |   0
 .../pulsar/client/impl/schema/AvroSchema.java  |   0
 .../pulsar/client/impl/schema/BytesSchema.java |   0
 .../pulsar/client/impl/schema/JSONSchema.java  |   0
 .../pulsar/client/impl/schema/ProtobufSchema.java  |   0
 .../pulsar/client/impl/schema/StringSchema.java|   0
 .../pulsar/client/schema}/AvroSchemaTest.java  |   4 +-
 .../pulsar/client/schema}/JSONSchemaTest.java  |   4 +-
 .../pulsar/client/schema}/ProtobufSchemaTest.java  |   8 +-
 .../src/test/proto/Test.proto  |   2 +-
 pulsar-client-shaded/pom.xml   |  32 +-
 pulsar-client/pom.xml  |  63 +---
 .../{schemas => schema}/DefaultSchemasTest.java|  16 +-
 19 files changed, 403 insertions(+), 158 deletions(-)

diff --git a/pom.xml b/pom.xml
index 2275471..a6d6453 100644
--- a/pom.xml
+++ b/pom.xml
@@ -82,6 +82,7 @@ flexible messaging model and an intuitive client 
API.
 pulsar-broker-shaded
 pulsar-client
 pulsar-client-shaded
+pulsar-client-schema
 pulsar-client-admin
 pulsar-client-admin-shaded
 pulsar-client-tools
diff --git a/pulsar-broker/pom.xml b/pulsar-broker/pom.xml
index ccee189..d505c7a 100644
--- a/pulsar-broker/pom.xml
+++ b/pulsar-broker/pom.xml
@@ -181,6 +181,11 @@
 
 
 
+  com.fasterxml.jackson.module
+  jackson-module-jsonSchema
+
+
+
   org.slf4j
   jul-to-slf4j
 
diff --git a/pulsar-client-admin-shaded/pom.xml 
b/pulsar-client-admin-shaded/pom.xml
index 2edc7cf..0c9fb93 100644
--- a/pulsar-client-admin-shaded/pom.xml
+++ b/pulsar-client-admin-shaded/pom.xml
@@ -98,15 +98,6 @@
   io.opencensus:*
   org.objenesis:*
   org.yaml:snakeyaml
-
-  org.apache.avro:*
-  
-  org.codehaus.jackson:jackson-core-asl
-  org.codehaus.jackson:jackson-mapper-asl
-  com.thoughtworks.paranamer:paranamer
-  org.xerial.snappy:snappy-java
-  org.apache.commons:commons-compress
-  org.tukaani:xz
 
   
   
@@ -211,27 +202,6 @@
   
org.apache.pulsar.shade.org.reactivestreams
 
   
-  org.apache.avro
-  
org.apache.pulsar.shade.org.apache.avro
-  
-  
-  
-  org.codehaus.jackson
-  
org.apache.pulsar.shade.org.codehaus.jackson
-  
-  
-  com.thoughtworks.paranamer
-  
org.apache.pulsar.shade.com.thoughtworks.paranamer
-  
-  
-  org.xerial.snappy
-  
org.apache.pulsar.shade.org.xerial.snappy
-  
-  
-  org.tukaani
-  
org.apache.pulsar.shade.org.tukaani
-  
-  
   io.grpc
   
org.apache.pulsar.shade.io.grpc
   
diff --git a/pulsar-client-kafka-compat/pulsar-client-kafka-shaded/pom.xml 
b/pulsar-client-kafka-compat/pulsar-client-kafka-shaded/pom.xml
index 5d474ba..3ba94e9 100644
--- a/pulsar-client-kafka-compat/pulsar-client-kafka-shaded/pom.xml
+++ b/pulsar-client-kafka-compat/pulsar-client-kafka-shaded/pom.xml
@@ -68,6 +68,7 @@
   org.apache.kafka:kafka-clients
   
org.apache.pulsar:pulsar-client-kafka-original
   org.apache.pulsar:pulsar-client-original
+  org.apache.pulsar:pulsar-client-schema
   org.apache.commons:commons-lang3
   commons-codec:commons-codec
   commons-collections:commons-collections
@@ -97,15 +98,6 @@
   commons-*:*
   org.yaml:snakeyaml
   org.obje

[GitHub] jerrypeng closed pull request #2168: fixing protobuf schema shading issue

2018-07-16 Thread GitBox

jerrypeng closed pull request #2168:  fixing protobuf schema shading issue
URL: https://github.com/apache/incubator-pulsar/pull/2168
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/pom.xml b/pom.xml
index 8a0da985ee..881b628883 100644
--- a/pom.xml
+++ b/pom.xml
@@ -82,6 +82,7 @@ flexible messaging model and an intuitive client 
API.
 pulsar-broker-shaded
 pulsar-client
 pulsar-client-shaded
+pulsar-client-schema
 pulsar-client-admin
 pulsar-client-admin-shaded
 pulsar-client-tools
diff --git a/pulsar-broker/pom.xml b/pulsar-broker/pom.xml
index ccee189a9c..d505c7a881 100644
--- a/pulsar-broker/pom.xml
+++ b/pulsar-broker/pom.xml
@@ -180,6 +180,11 @@
   jackson-jaxrs-json-provider
 
 
+
+  com.fasterxml.jackson.module
+  jackson-module-jsonSchema
+
+
 
   org.slf4j
   jul-to-slf4j
diff --git a/pulsar-client-admin-shaded/pom.xml 
b/pulsar-client-admin-shaded/pom.xml
index 2edc7cf971..0c9fb93130 100644
--- a/pulsar-client-admin-shaded/pom.xml
+++ b/pulsar-client-admin-shaded/pom.xml
@@ -98,15 +98,6 @@
   io.opencensus:*
   org.objenesis:*
   org.yaml:snakeyaml
-
-  org.apache.avro:*
-  
-  org.codehaus.jackson:jackson-core-asl
-  org.codehaus.jackson:jackson-mapper-asl
-  com.thoughtworks.paranamer:paranamer
-  org.xerial.snappy:snappy-java
-  org.apache.commons:commons-compress
-  org.tukaani:xz
 
   
   
@@ -210,27 +201,6 @@
   org.reactivestreams
   
org.apache.pulsar.shade.org.reactivestreams
 
-  
-  org.apache.avro
-  
org.apache.pulsar.shade.org.apache.avro
-  
-  
-  
-  org.codehaus.jackson
-  
org.apache.pulsar.shade.org.codehaus.jackson
-  
-  
-  com.thoughtworks.paranamer
-  
org.apache.pulsar.shade.com.thoughtworks.paranamer
-  
-  
-  org.xerial.snappy
-  
org.apache.pulsar.shade.org.xerial.snappy
-  
-  
-  org.tukaani
-  
org.apache.pulsar.shade.org.tukaani
-  
   
   io.grpc
   
org.apache.pulsar.shade.io.grpc
diff --git a/pulsar-client-kafka-compat/pulsar-client-kafka-shaded/pom.xml 
b/pulsar-client-kafka-compat/pulsar-client-kafka-shaded/pom.xml
index 5d474ba66e..3ba94e9aaf 100644
--- a/pulsar-client-kafka-compat/pulsar-client-kafka-shaded/pom.xml
+++ b/pulsar-client-kafka-compat/pulsar-client-kafka-shaded/pom.xml
@@ -68,6 +68,7 @@
   org.apache.kafka:kafka-clients
   
org.apache.pulsar:pulsar-client-kafka-original
   org.apache.pulsar:pulsar-client-original
+  org.apache.pulsar:pulsar-client-schema
   org.apache.commons:commons-lang3
   commons-codec:commons-codec
   commons-collections:commons-collections
@@ -97,15 +98,6 @@
   commons-*:*
   org.yaml:snakeyaml
   org.objenesis:*
-
-  org.apache.avro:*
-  
-  org.codehaus.jackson:jackson-core-asl
-  org.codehaus.jackson:jackson-mapper-asl
-  com.thoughtworks.paranamer:paranamer
-  org.xerial.snappy:snappy-java
-  org.apache.commons:commons-compress
-  org.tukaani:xz
 
   

@@ -188,27 +180,6 @@
   org.apache.http
   
org.apache.pulsar.shade.org.apache.http
 
-
-  org.apache.avro
-  
org.apache.pulsar.shade.org.apache.avro
-
-
-
-  org.codehaus.jackson
-  
org.apache.pulsar.shade.org.codehaus.jackson
-
-
-  com.thoughtworks.paranamer
-  
org.apache.pulsar.shade.com.thoughtworks.paranamer
-
-
-  org.xerial.snappy
-  
org.apache.pulsar.shade.org.xerial.snappy
-
-
-  org.tukaani
-  
org.apache.pulsar.shade.org.tukaani
-

[GitHub] ivankelly commented on issue #2172: If auth is badly configured in c++, close the connection

2018-07-16 Thread GitBox

ivankelly commented on issue #2172: If auth is badly configured in c++, close 
the connection
URL: https://github.com/apache/incubator-pulsar/pull/2172#issuecomment-405279994
 
 
   @merlimat this is the segfault I was talking about last week. Functions is 
using the factory to create auth. This PR fixes the segault, but it'll still 
fail. I'll have another patch for actually making it work.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivankelly opened a new pull request #2172: If auth is badly configured in c++, close the connection

2018-07-16 Thread GitBox

ivankelly opened a new pull request #2172: If auth is badly configured in c++, 
close the connection
URL: https://github.com/apache/incubator-pulsar/pull/2172
 
 
   This patch fixes a segfault where, if auth was badly configured,
   resulting in a null authentication pointer, the client would still try
   to use it to connect, and as a result, segfault.
   
   The patch adds a null check on the authentication object, and a check
   that the connection object is still valid before initialization of the
   tcp channel.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-pulsar] branch asf-site updated: Updated site at revision 555dd57

2018-07-16 Thread mmerli

This is an automated email from the ASF dual-hosted git repository.

mmerli pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new a2df34d  Updated site at revision 555dd57
a2df34d is described below

commit a2df34d59d5f650ff05776db99b9818574b8e12b
Author: jenkins 
AuthorDate: Mon Jul 16 09:37:03 2018 +

Updated site at revision 555dd57
---
 .../docs/latest/adaptors/PulsarSpark/index.html| 10 +-
 .../docs/latest/adaptors/PulsarStorm/index.html|  8 
 content/docs/latest/admin-api/overview/index.html  | 18 +-
 content/docs/latest/clients/Cpp/index.html |  8 
 content/docs/latest/clients/Java/index.html| 16 
 content/docs/latest/clients/Python/index.html  | 10 +-
 content/docs/latest/clients/WebSocket/index.html   |  8 
 content/docs/latest/clients/go/index.html  |  6 +++---
 .../docs/latest/cookbooks/Encryption/index.html|  6 +++---
 .../latest/cookbooks/PartitionedTopics/index.html  | 16 
 .../latest/cookbooks/RetentionExpiry/index.html| 16 
 .../docs/latest/cookbooks/compaction/index.html|  8 
 .../cookbooks/message-deduplication/index.html | 12 ++--
 .../docs/latest/cookbooks/message-queue/index.html | 16 
 .../latest/cookbooks/tiered-storage/index.html |  6 +++---
 .../docs/latest/deployment/Kubernetes/index.html   |  4 ++--
 .../docs/latest/deployment/aws-cluster/index.html  |  6 +++---
 content/docs/latest/deployment/cluster/index.html  |  6 +++---
 content/docs/latest/deployment/instance/index.html |  6 +++---
 .../ConceptsAndArchitecture/index.html |  4 ++--
 .../latest/getting-started/LocalCluster/index.html |  4 ++--
 .../latest/getting-started/Pulsar-2.0/index.html   |  2 +-
 .../docs/latest/getting-started/docker/index.html  |  4 ++--
 .../docs/latest/project/BinaryProtocol/index.html  |  4 ++--
 content/docs/latest/project/CompileCpp/index.html  |  8 
 .../docs/latest/project/SimulationTools/index.html |  2 +-
 .../docs/latest/project/schema-storage/index.html  |  4 ++--
 content/docs/latest/reference/CliTools/index.html  | 22 +++---
 content/docs/latest/reference/RestApi/index.html   |  4 ++--
 .../docs/latest/security/authorization/index.html  | 14 +++---
 content/docs/latest/security/encryption/index.html |  6 +++---
 content/ja/adaptors/PulsarSpark/index.html |  8 
 content/ja/adaptors/PulsarStorm/index.html |  6 +++---
 content/ja/admin/AdminInterface/index.html | 12 ++--
 content/ja/admin/Authz/index.html  | 12 ++--
 content/ja/admin/ClustersBrokers/index.html|  6 +++---
 content/ja/admin/PropertiesNamespaces/index.html   |  6 +++---
 content/ja/advanced/PartitionedTopics/index.html   | 12 ++--
 content/ja/advanced/RetentionExpiry/index.html | 12 ++--
 content/ja/clients/Cpp/index.html  |  6 +++---
 content/ja/clients/Java/index.html |  8 
 content/ja/clients/Python/index.html   |  8 
 content/ja/clients/WebSocket/index.html|  8 
 content/ja/deployment/InstanceSetup/index.html |  6 +++---
 content/ja/deployment/Kubernetes/index.html|  4 ++--
 .../ConceptsAndArchitecture/index.html |  2 +-
 content/ja/getting-started/LocalCluster/index.html |  4 ++--
 content/ja/project/BinaryProtocol/index.html   |  4 ++--
 content/ja/project/SimulationTools/index.html  |  2 +-
 content/ja/reference/CliTools/index.html   | 18 +-
 content/ja/reference/RestApi/index.html|  4 ++--
 51 files changed, 206 insertions(+), 206 deletions(-)

diff --git a/content/docs/latest/adaptors/PulsarSpark/index.html 
b/content/docs/latest/adaptors/PulsarSpark/index.html
index 396ae0c..ae1dfd8 100644
--- a/content/docs/latest/adaptors/PulsarSpark/index.html
+++ b/content/docs/latest/adaptors/PulsarSpark/index.html
@@ -1086,9 +1086,9 @@
   
   
   
+  Spark Streaming 
Pulsar receiver
   
   
-  Spark Streaming 
Pulsar receiver
   
   
   
@@ -1330,9 +1330,9 @@
   
   
   
+  Spark Streaming 
Pulsar receiver
   
   
-  Spark Streaming 
Pulsar receiver
   
   
   
@@ -1540,8 +1540,6 @@
   
   
   
-  
-  
   The Pulsar Java 
client
   
   
@@ -1620,6 +1618,8 @@
   
   
   
+  
+  
   Authentication 
and authorization in Pulsar
   
   
@@ -1672,11 +1672,11 @@
   
   
   
-  The Pulsar admin 
interface

[GitHub] sijie commented on issue #2171: Add integration test for kafka source

2018-07-16 Thread GitBox

sijie commented on issue #2171: Add integration test for kafka source
URL: https://github.com/apache/incubator-pulsar/pull/2171#issuecomment-405167056
 
 
   This change is based on #2170 . The git commit 52f8c74 is the one to review 
for this PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie opened a new pull request #2171: Add integration test for kafka source

2018-07-16 Thread GitBox

sijie opened a new pull request #2171: Add integration test for kafka source
URL: https://github.com/apache/incubator-pulsar/pull/2171
 
 
   *Motivation*
   
   We added integration tests for kafka & cassandra sinks. We need test 
coverage on kafka sources.
   
   *Changes*
   
   - Add `PulsarIOSourceTest` and `SourceTester` for testing sources
   - Implement `KafkaSourceTester` for testing kafka source


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

67 matches

Mail list logo