[GitHub] [pulsar] jiazhai opened a new issue #3777: Documentation: link in standalone page failed
jiazhai opened a new issue #3777: Documentation: link in standalone page failed URL: https://github.com/apache/pulsar/issues/3777 http://pulsar.apache.org/docs/en/standalone/ ![image](https://user-images.githubusercontent.com/6348208/53940813-94d82500-40f1-11e9-8648-466d5e344143.png) This is because in 2.3.0 each connector is packaged into a separate nar. We need to change the link from old: `https://archive.apache.org/dist/pulsar/pulsar-2.3.0/apache-pulsar-io-connectors-2.3.0-bin.tar.gz` into `https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=pulsar/pulsar-2.3.0/connectors/` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] srkukarni merged pull request #3722: Retry creation of assignment topic a few times before giving up
srkukarni merged pull request #3722: Retry creation of assignment topic a few times before giving up URL: https://github.com/apache/pulsar/pull/3722 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch retry_creation deleted (was 024831f)
This is an automated email from the ASF dual-hosted git repository. sanjeevrk pushed a change to branch retry_creation in repository https://gitbox.apache.org/repos/asf/pulsar.git. was 024831f Merge branch 'master' into retry_creation The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[GitHub] [pulsar] srkukarni commented on issue #3717: Expose Secrets in Source/SinkContext
srkukarni commented on issue #3717: Expose Secrets in Source/SinkContext URL: https://github.com/apache/pulsar/pull/3717#issuecomment-470417529 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated: Retry creation of assignment topic a few times before giving up (#3722)
This is an automated email from the ASF dual-hosted git repository. sanjeevrk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new fd05219 Retry creation of assignment topic a few times before giving up (#3722) fd05219 is described below commit fd05219a3593fdb69e2efd71bed3e5895be91ea8 Author: Sanjeev Kulkarni AuthorDate: Wed Mar 6 23:31:40 2019 -0800 Retry creation of assignment topic a few times before giving up (#3722) * Retry creation of assignment topic a few times before giving up * Use Action based retry mechanism * Fix build * Catch interrupted exception * Fix unittest * Added header --- .../functions/runtime/KubernetesRuntime.java | 87 ++--- .../pulsar/functions/runtime/RuntimeUtils.java | 105 .../org/apache/pulsar/functions/utils/Actions.java | 139 + .../pulsar/functions/utils/ActionsTest.java} | 70 ++- .../pulsar/functions/worker/FunctionActioner.java | 14 +-- .../pulsar/functions/worker/SchedulerManager.java | 71 ++- 6 files changed, 264 insertions(+), 222 deletions(-) diff --git a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/KubernetesRuntime.java b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/KubernetesRuntime.java index 3fc0c69..3c0468d 100644 --- a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/KubernetesRuntime.java +++ b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/KubernetesRuntime.java @@ -60,6 +60,7 @@ import org.apache.pulsar.functions.proto.InstanceCommunication; import org.apache.pulsar.functions.proto.InstanceCommunication.FunctionStatus; import org.apache.pulsar.functions.proto.InstanceControlGrpc; import org.apache.pulsar.functions.secretsproviderconfigurator.SecretsProviderConfigurator; +import org.apache.pulsar.functions.utils.Actions; import org.apache.pulsar.functions.utils.FunctionDetailsUtils; import org.apache.pulsar.functions.utils.Utils; @@ -360,7 +361,7 @@ public class KubernetesRuntime implements Runtime { String fqfn = FunctionDetailsUtils.getFullyQualifiedName(instanceConfig.getFunctionDetails()); -RuntimeUtils.Actions.Action createService = RuntimeUtils.Actions.Action.builder() +Actions.Action createService = Actions.Action.builder() .actionName(String.format("Submitting service for function %s", fqfn)) .numRetries(NUM_RETRIES) .sleepBetweenInvocationsMs(SLEEP_BETWEEN_RETRIES_MS) @@ -372,25 +373,25 @@ public class KubernetesRuntime implements Runtime { // already exists if (e.getCode() == HTTP_CONFLICT) { log.warn("Service already present for function {}", fqfn); -return RuntimeUtils.Actions.ActionResult.builder().success(true).build(); +return Actions.ActionResult.builder().success(true).build(); } String errorMsg = e.getResponseBody() != null ? e.getResponseBody() : e.getMessage(); -return RuntimeUtils.Actions.ActionResult.builder() +return Actions.ActionResult.builder() .success(false) .errorMsg(errorMsg) .build(); } -return RuntimeUtils.Actions.ActionResult.builder().success(true).build(); +return Actions.ActionResult.builder().success(true).build(); }) .build(); AtomicBoolean success = new AtomicBoolean(false); -RuntimeUtils.Actions.newBuilder() +Actions.newBuilder() .addAction(createService.toBuilder() -.onSuccess(() -> success.set(true)) +.onSuccess((ignored) -> success.set(true)) .build()) .run(); @@ -432,7 +433,7 @@ public class KubernetesRuntime implements Runtime { String fqfn = FunctionDetailsUtils.getFullyQualifiedName(instanceConfig.getFunctionDetails()); -RuntimeUtils.Actions.Action createStatefulSet = RuntimeUtils.Actions.Action.builder() +Actions.Action createStatefulSet = Actions.Action.builder() .actionName(String.format("Submitting statefulset for function %s", fqfn)) .numRetries(NUM_RETRIES) .sleepBetweenInvocationsMs(SLEEP_BETWEEN_RETRIES_MS) @@ -444,25 +445,25 @@ public class KubernetesRuntime implements Runtime { // already exists if (e.getCode() == HTTP_CONFLICT) {
[GitHub] [pulsar] merlimat commented on issue #3671: pulsar-admin topics list public/default fails
merlimat commented on issue #3671: pulsar-admin topics list public/default fails URL: https://github.com/apache/pulsar/issues/3671#issuecomment-470415406 Fix is in #3776 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager…
merlimat commented on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager… URL: https://github.com/apache/pulsar/pull/3760#issuecomment-470412358 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] skyrocknroll commented on issue #3766: Support passing schema definition for JSON and AVRO schemas
skyrocknroll commented on issue #3766: Support passing schema definition for JSON and AVRO schemas URL: https://github.com/apache/pulsar/pull/3766#issuecomment-470411918 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] skyrocknroll opened a new issue #3671: pulsar-admin topics list public/default fails
skyrocknroll opened a new issue #3671: pulsar-admin topics list public/default fails URL: https://github.com/apache/pulsar/issues/3671 **Describe the bug** pulsar-admin topics list public/default Os Ubuntu 16 04 Pulsar 2.3.0 Failed with following error in log ``` [2019-02-22T13:59:00.704Z] INFO pulsar-web-29-1 RequestLog:60 10.10.58.5 - - [22/Feb/2019:13:59:00 +] "GET /admin/clusters/stage HTTP/1.1" 200 224 "-" "python-requests/2.19.1" 2 [2019-02-22T13:59:00.746Z] INFO pulsar-web-29-3 RequestLog:60 10.10.58.5 - - [22/Feb/2019:13:59:00 +] "GET /admin/broker-stats/destinations HTTP/1.1" 200 1266 "-" "python-requests/2.19.1" 2 [2019-02-22T13:59:05.706Z] INFO pulsar-web-29-4 RequestLog:60 10.10.58.5 - - [22/Feb/2019:13:59:05 +] "GET /admin/clusters/stage HTTP/1.1" 200 224 "-" "python-requests/2.19.1" 2 [2019-02-22T13:59:05.750Z] INFO pulsar-web-29-3 RequestLog:60 10.10.58.5 - - [22/Feb/2019:13:59:05 +] "GET /admin/broker-stats/destinations HTTP/1.1" 200 1266 "-" "python-requests/2.19.1" 1 [2019-02-22T13:59:06.354Z] INFO pulsar-web-29-1 RequestLog:60 127.0.0.1 - - [22/Feb/2019:13:59:06 +] "GET /admin/v2/persistent/public/default HTTP/1.1" 200 396 "-" "Jersey/2.27 (HttpUrlConnection 1.8.0_201)" 8 [2019-02-22T14:00:06.407Z] INFO pulsar-web-29-4 RequestLog:60 10.10.58.5 - - [22/Feb/2019:14:00:06 +] "GET /admin/brokers/stage HTTP/1.1" 200 91 "-" "python-requests/2.19.1" 3 [2019-02-22T14:01:01.902Z] INFO pulsar-ordered-OrderedExecutor-5-0-EventThread ZooKeeperDataCache:153 [State:CONNECTED Timeout:3 sessionid:0x1049b573fba001a local:/10.10.58.7:39486 remoteserver:zk-node-1.bm.infra/10.10.58.15:2181 lastZxid:4294968305 xid:532 sent:532 recv:538 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: WatchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/pulsar-node-1.bm.infra:8080 [2019-02-22T14:01:06.526Z] ERROR pulsar-web-29-3 NonPersistentTopics:198 [null] Failed to get list of topics under namespace public/default java.util.concurrent.ExecutionException: org.apache.pulsar.client.admin.PulsarAdminException: java.net.SocketTimeoutException: Read timed out at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_201] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[?:1.8.0_201] at org.apache.pulsar.broker.admin.v2.NonPersistentTopics.getList(NonPersistentTopics.java:186) ~[org.apache.pulsar-pulsar-broker-2.3.0.jar:2.3.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_201] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) ~[org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:243) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) [org.glassfish.jersey.core-jersey-common-2.27.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) [org.glassfish.jersey.core-jersey-common-2.27.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:316) [org.glassfish.jersey.core-jersey-common-2
[GitHub] [pulsar] skyrocknroll commented on issue #3671: pulsar-admin topics list public/default fails
skyrocknroll commented on issue #3671: pulsar-admin topics list public/default fails URL: https://github.com/apache/pulsar/issues/3671#issuecomment-470411519 It was a dev cluster. For some reason, we redeployed the whole cluster and the issue went away This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #3677: PIP-30: interface and mutual change authentication
jiazhai commented on issue #3677: PIP-30: interface and mutual change authentication URL: https://github.com/apache/pulsar/pull/3677#issuecomment-470408233 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] srkukarni commented on issue #3717: Expose Secrets in Source/SinkContext
srkukarni commented on issue #3717: Expose Secrets in Source/SinkContext URL: https://github.com/apache/pulsar/pull/3717#issuecomment-470404482 run integration tests run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] srkukarni commented on issue #3770: Allow users to update everything in inputspecs except for isregexpattern
srkukarni commented on issue #3770: Allow users to update everything in inputspecs except for isregexpattern URL: https://github.com/apache/pulsar/pull/3770#issuecomment-470404385 run integration tests run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] MarvinCai commented on issue #3753: Add support for Kafka's ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG (#1090)
MarvinCai commented on issue #3753: Add support for Kafka's ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG (#1090) URL: https://github.com/apache/pulsar/pull/3753#issuecomment-470402482 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #3722: Retry creation of assignment topic a few times before giving up
jerrypeng commented on issue #3722: Retry creation of assignment topic a few times before giving up URL: https://github.com/apache/pulsar/pull/3722#issuecomment-470391276 rerun cpp tests rerun integration tests rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #3737: [python client] set protobuf version to be larger than 3.6.0
jerrypeng commented on issue #3737: [python client] set protobuf version to be larger than 3.6.0 URL: https://github.com/apache/pulsar/pull/3737#issuecomment-470391068 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #3740: fix document description bugs
jerrypeng commented on issue #3740: fix document description bugs URL: https://github.com/apache/pulsar/pull/3740#issuecomment-470391019 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #3739: [Issue #3728] Add several instructions for IO Connectors
jerrypeng commented on issue #3739: [Issue #3728] Add several instructions for IO Connectors URL: https://github.com/apache/pulsar/pull/3739#issuecomment-470390925 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #3735: Implementing authentication for Pulsar Functions
jerrypeng commented on issue #3735: Implementing authentication for Pulsar Functions URL: https://github.com/apache/pulsar/pull/3735#issuecomment-470390768 @merlimat @sijie @srkukarni @jiazhai please review. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #3747: Closing FunctionAssignmentTailer should handle ConsumerAlreadyClosedException
jerrypeng commented on issue #3747: Closing FunctionAssignmentTailer should handle ConsumerAlreadyClosedException URL: https://github.com/apache/pulsar/pull/3747#issuecomment-470390548 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #3776: Use at least 8 threads in Jetty thread pool
jerrypeng commented on issue #3776: Use at least 8 threads in Jetty thread pool URL: https://github.com/apache/pulsar/pull/3776#issuecomment-470390150 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] MarvinCai commented on issue #3753: Add support for Kafka's ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG (#1090)
MarvinCai commented on issue #3753: Add support for Kafka's ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG (#1090) URL: https://github.com/apache/pulsar/pull/3753#issuecomment-470387565 rerun java8 tests rerun integration tests rerun cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] lxkaka commented on issue #3671: pulsar-admin topics list public/default fails
lxkaka commented on issue #3671: pulsar-admin topics list public/default fails URL: https://github.com/apache/pulsar/issues/3671#issuecomment-470385603 I have the same problem. `pulsar-admin topics list public/default` or `pulsar-admin persistent list public/default` show error `null Reason: javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: Read timed out` Os Ubuntu 16 04 Pulsar 2.3.0 broker log output just same as @skyrocknroll posted ``` 12:57:55.146 [pulsar-web-28-2] ERROR org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Failed to get list of topics under namespace public/default java.util.concurrent.ExecutionException: org.apache.pulsar.client.admin.PulsarAdminException: java.net.SocketTimeoutException: Read timed out at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_191] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[?:1.8.0_191] at org.apache.pulsar.broker.admin.v2.NonPersistentTopics.getList(NonPersistentTopics.java:186) ~[org.apache.pulsar-pulsar-broker-2.3.0.jar:2.3.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_191] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_191] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_191] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) ~[org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:243) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) [org.glassfish.jersey.core-jersey-common-2.27.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) [org.glassfish.jersey.core-jersey-common-2.27.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:316) [org.glassfish.jersey.core-jersey-common-2.27.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:298) [org.glassfish.jersey.core-jersey-common-2.27.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:268) [org.glassfish.jersey.core-jersey-common-2.27.jar:?] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) [org.glassfish.jersey.core-jersey-common-2.27.jar:?] at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) [org.glassfish.jersey.core-jersey-server-2.27.jar:?] at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?] at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?] at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?] at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?] at org.glassfish.jersey.servlet.ServletContainer.s
[GitHub] [pulsar] merlimat opened a new pull request #3776: Use at least 8 threads in Jetty thread pool
merlimat opened a new pull request #3776: Use at least 8 threads in Jetty thread pool URL: https://github.com/apache/pulsar/pull/3776 ### Motivation Use at least 8 threads to avoid having Jetty go into threads starving and having the possibility of getting into a deadlock where a Jetty thread is waiting for another HTTP call to complete in same thread. This solve the issues of requests timing out when the broker is making REST calls to itself. Such examples are when running in standalone mode and creating a function. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager…
merlimat commented on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager… URL: https://github.com/apache/pulsar/pull/3760#issuecomment-470378274 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat merged pull request #3761: Updated consumer example to use negative acks
merlimat merged pull request #3761: Updated consumer example to use negative acks URL: https://github.com/apache/pulsar/pull/3761 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #3703: Added support for "negative acks" in Java client
merlimat commented on issue #3703: Added support for "negative acks" in Java client URL: https://github.com/apache/pulsar/pull/3703#issuecomment-470375735 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated: Updated consumer example to use negative acks (#3761)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 17b531b Updated consumer example to use negative acks (#3761) 17b531b is described below commit 17b531bd18b2e00714d0b2e4bc8d6cfdc2dc1647 Author: Matteo Merli AuthorDate: Wed Mar 6 19:58:47 2019 -0800 Updated consumer example to use negative acks (#3761) --- site2/docs/client-libraries-go.md | 16 +--- site2/docs/client-libraries-java.md | 19 +-- site2/docs/client-libraries-python.md | 18 ++ site2/docs/reference-terminology.md | 7 ++- 4 files changed, 46 insertions(+), 14 deletions(-) diff --git a/site2/docs/client-libraries-go.md b/site2/docs/client-libraries-go.md index 5d7466b..5e22ac6 100644 --- a/site2/docs/client-libraries-go.md +++ b/site2/docs/client-libraries-go.md @@ -262,7 +262,9 @@ Method | Description | Return type `Receive(context.Context)` | Receives a single message from the topic. This method blocks until a message is available. | `(Message, error)` `Ack(Message)` | [Acknowledges](reference-terminology.md#acknowledgment-ack) a message to the Pulsar [broker](reference-terminology.md#broker) | `error` `AckID(MessageID)` | [Acknowledges](reference-terminology.md#acknowledgment-ack) a message to the Pulsar [broker](reference-terminology.md#broker) by message ID | `error` -`AckCumulative(Message)` | [Acknowledges](reference-terminology.md#acknowledgment-ack) *all* the messages in the stream, up to and including the specified message. The `AckCumulative` method will block until the ack has been sent to the broker. After that, the messages will *not* be redelivered to the consumer. Cumulative acking can only be used with a [shared](concepts-messaging.md#shared) subscription type. +`AckCumulative(Message)` | [Acknowledges](reference-terminology.md#acknowledgment-ack) *all* the messages in the stream, up to and including the specified message. The `AckCumulative` method will block until the ack has been sent to the broker. After that, the messages will *not* be redelivered to the consumer. Cumulative acking can only be used with a [shared](concepts-messaging.md#shared) subscription type. | `error` +`Nack(Message)` | Acknowledge the failure to process a single message. | `error` +`NackID(MessageID)` | Acknowledge the failure to process a single message. | `error` `Close()` | Closes the consumer, disabling its ability to receive messages from the broker | `error` `RedeliverUnackedMessages()` | Redelivers *all* unacknowledged messages on the topic. In [failover](concepts-messaging.md#failover) mode, this request is ignored if the consumer isn't active on the specified topic; in [shared](concepts-messaging.md#shared) mode, redelivered messages are distributed across all consumers connected to the topic. **Note**: this is a *non-blocking* operation that doesn't throw an error. | @@ -305,8 +307,15 @@ func main() { if err != nil { log.Fatal(err) } // Do something with the message - -consumer.Ack(msg) +err = processMessage(msg) + +if err == nil { +// Message processed successfully +consumer.Ack(msg) +} else { +// Failed to process messages +consumer.Nack(msg) +} } } ``` @@ -319,6 +328,7 @@ Parameter | Description | Default `SubscriptionName` | The subscription name for this consumer | `Name` | The name of the consumer | `AckTimeout` | | 0 +`NackRedeliveryDelay` | The delay after which to redeliver the messages that failed to be processed. Default is 1min. (See `Consumer.Nack()`) | 1 minute `SubscriptionType` | Available options are `Exclusive`, `Shared`, and `Failover` | `Exclusive` `MessageChannel` | The Go channel used by the consumer. Messages that arrive from the Pulsar topic(s) will be passed to this channel. | `ReceiverQueueSize` | Sets the size of the consumer's receiver queue, i.e. the number of messages that can be accumulated by the consumer before the application calls `Receive`. A value higher than the default of 1000 could increase consumer throughput, though at the expense of more memory utilization. | 1000 diff --git a/site2/docs/client-libraries-java.md b/site2/docs/client-libraries-java.md index b588e0f..b8680d8 100644 --- a/site2/docs/client-libraries-java.md +++ b/site2/docs/client-libraries-java.md @@ -186,18 +186,25 @@ Consumer consumer = client.newConsumer() .subscribe(); ``` -The `subscribe` method will automatically subscribe the consumer to the specified topic and subscription. One way to make the consumer listen on the topic is to set up a `while` loop. In this example loop, the consumer listens for messages, prints the contents of any message that's received, and then [acknowledges](reference-
[GitHub] [pulsar] david-streamlio opened a new pull request #3775: Pulsar-IO: Added Kinesis Source
david-streamlio opened a new pull request #3775: Pulsar-IO: Added Kinesis Source URL: https://github.com/apache/pulsar/pull/3775 ### Motivation Added an AWS Kinesis Source connector to the existing Kinesis module. ### Modifications - Added an AWS Kinesis Source connector and associated configuration class - Added test class for AWS Kinesis Source connector - Refactored Configuration hierarchy to promote code reused between the Sink and Source configurations. - Added test cases to confirm the above refactoring did NOT break anything in the Sink ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change added tests and can be verified as follows: - Added unit tests ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): NO - The public API: NO - The schema: NO - The default values of configurations: NO - The wire protocol: NO - The rest endpoints: NO - The admin cli options: NO - Anything that affects deployment: NO ### Documentation - Does this pull request introduce a new feature? YES - If yes, how is the feature documented? YES, I updated the docs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] MarvinCai commented on a change in pull request #3753: Add support for Kafka's ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG (#1090)
MarvinCai commented on a change in pull request #3753: Add support for Kafka's ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG (#1090) URL: https://github.com/apache/pulsar/pull/3753#discussion_r263226100 ## File path: pulsar-client-kafka-compat/pulsar-client-kafka/src/main/java/org/apache/kafka/clients/producer/PulsarKafkaProducer.java ## @@ -34,6 +34,7 @@ import java.util.concurrent.TimeoutException; import java.util.stream.Collectors; +import com.sun.javaws.exceptions.InvalidArgumentException; Review comment: Removed unused import. Also added document that merlimat mentioned. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] MarvinCai commented on issue #3753: Add support for Kafka's ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG (#1090)
MarvinCai commented on issue #3753: Add support for Kafka's ProducerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG (#1090) URL: https://github.com/apache/pulsar/pull/3753#issuecomment-470370713 rerun java8 tests rerun integration tests rerun cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy commented on issue #3764: Fixed Reader.HasNext() in Go client
wolfstudy commented on issue #3764: Fixed Reader.HasNext() in Go client URL: https://github.com/apache/pulsar/pull/3764#issuecomment-470360644 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #3535: Deduplication causes a lot of Full GCs
merlimat commented on issue #3535: Deduplication causes a lot of Full GCs URL: https://github.com/apache/pulsar/issues/3535#issuecomment-470360296 Updated #3612 with correct fix This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie commented on issue #3761: Updated consumer example to use negative acks
sijie commented on issue #3761: Updated consumer example to use negative acks URL: https://github.com/apache/pulsar/pull/3761#issuecomment-470359470 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy commented on a change in pull request #3759: Fixes for Go client unit test
wolfstudy commented on a change in pull request #3759: Fixes for Go client unit test URL: https://github.com/apache/pulsar/pull/3759#discussion_r263216289 ## File path: pulsar-client-go/pulsar/consumer_test.go ## @@ -98,7 +100,10 @@ func TestConsumer(t *testing.T) { assert.NotNil(t, msg) assert.Equal(t, string(msg.Payload()), fmt.Sprintf("hello-%d", i)) - assert.Equal(t, string(msg.Topic()), "persistent://public/default/my-topic") + assert.Equal(t, string(msg.Topic()), "persistent://public/default/" + topic) Review comment: ```suggestion assert.Equal(t, msg.Topic(), "persistent://public/default/" + topic) ``` The type of `msg.Topic` itself is string, we don't need to convert here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy commented on a change in pull request #3759: Fixes for Go client unit test
wolfstudy commented on a change in pull request #3759: Fixes for Go client unit test URL: https://github.com/apache/pulsar/pull/3759#discussion_r263216289 ## File path: pulsar-client-go/pulsar/consumer_test.go ## @@ -98,7 +100,10 @@ func TestConsumer(t *testing.T) { assert.NotNil(t, msg) assert.Equal(t, string(msg.Payload()), fmt.Sprintf("hello-%d", i)) - assert.Equal(t, string(msg.Topic()), "persistent://public/default/my-topic") + assert.Equal(t, string(msg.Topic()), "persistent://public/default/" + topic) Review comment: ```suggestion assert.Equal(t, msg.Topic(), "persistent://public/default/" + topic) ``` The type of msg.Topic itself is string, we don't need to convert here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] rdhabalia removed a comment on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager…
rdhabalia removed a comment on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager… URL: https://github.com/apache/pulsar/pull/3760#issuecomment-470356454 rerun java8 tests rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] rdhabalia commented on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager…
rdhabalia commented on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager… URL: https://github.com/apache/pulsar/pull/3760#issuecomment-470356454 rerun java8 tests rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy commented on issue #3759: Fixes for Go client unit test
wolfstudy commented on issue #3759: Fixes for Go client unit test URL: https://github.com/apache/pulsar/pull/3759#issuecomment-470356154 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #3612: Set the dedup cursor as "inactive" after recovery
merlimat commented on issue #3612: Set the dedup cursor as "inactive" after recovery URL: https://github.com/apache/pulsar/pull/3612#issuecomment-470354878 @hrsakai @massakam Did more experimentation and fixed it by making sure the dedup cursor is never considered "active" in order to never trigger caching behavior. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] rdhabalia removed a comment on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager…
rdhabalia removed a comment on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager… URL: https://github.com/apache/pulsar/pull/3760#issuecomment-470334799 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy opened a new issue #3774: [go client] add go related tests and checks in jenkins
wolfstudy opened a new issue #3774: [go client] add go related tests and checks in jenkins URL: https://github.com/apache/pulsar/issues/3774 **Is your feature request related to a problem? Please describe.** At present, Jenkins does not check the unit tests and code specifications related to go. When we modify the unit test of go or add new code, we can only check it locally. For some code formats and specifications, we have no way. Do a good job of specification and unification **Describe the solution you'd like** Regarding the check of the go code format, the more mature tool in the community is gometalinter **Additional context** check.sh ``` #!/bin/bash # The script does automatic checking on a Go package and its sub-packages, including: # 1. gofmt (http://golang.org/cmd/gofmt/) # 2. golint(https://github.com/golang/lint) # 3. go vet(http://golang.org/cmd/vet) # 4. gosimple (https://github.com/dominikh/go-simple) # 5. unconvert (https://github.com/mdempsky/unconvert) # # gometalinter (github.com/alecthomas/gometalinter) is used to run each static # checker. set -ex # Make sure gometalinter is installed and $GOPATH/bin is in your path. # $ go get -v github.com/alecthomas/gometalinter" # $ gometalinter --install" if [ ! -x "$(type -p gometalinter.v2)" ]; then exit 1 fi linter_targets=$(go list ./...) # Automatic checks test -z "$(gometalinter.v2 -j 4 --disable-all \ --enable=gofmt \ --enable=golint \ --enable=vet \ --enable=gosimple \ --enable=unconvert \ --deadline=10m $linter_targets 2>&1 | grep -v 'ALL_CAPS\|OP_' 2>&1 | tee /dev/stderr)" GO111MODULE=on go test $linter_targets ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #3677: PIP-30: interface and mutual change authentication
jiazhai commented on issue #3677: PIP-30: interface and mutual change authentication URL: https://github.com/apache/pulsar/pull/3677#issuecomment-470347984 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] aahmed-se commented on issue #3772: Force before and after methods to always run
aahmed-se commented on issue #3772: Force before and after methods to always run URL: https://github.com/apache/pulsar/pull/3772#issuecomment-470345592 @rdhabalia it's one part of the problem , if one test fails teardown doesn't run so it won't solve the issue but it will the right logs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] lovelle commented on issue #3748: [cpp client] Bugfix prevent dup consumer for same topic subscription
lovelle commented on issue #3748: [cpp client] Bugfix prevent dup consumer for same topic subscription URL: https://github.com/apache/pulsar/pull/3748#issuecomment-470340551 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] lovelle commented on issue #3746: [java client] Bugfix prevent dup consumers for same topic subscription
lovelle commented on issue #3746: [java client] Bugfix prevent dup consumers for same topic subscription URL: https://github.com/apache/pulsar/pull/3746#issuecomment-470340371 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] rdhabalia commented on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager…
rdhabalia commented on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager… URL: https://github.com/apache/pulsar/pull/3760#issuecomment-470334799 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] rdhabalia commented on issue #3772: Force before and after methods to always run
rdhabalia commented on issue #3772: Force before and after methods to always run URL: https://github.com/apache/pulsar/pull/3772#issuecomment-470333240 how does it help if function-worker actually has the deadlock issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] aahmed-se commented on issue #3772: Force before and after methods to always run
aahmed-se commented on issue #3772: Force before and after methods to always run URL: https://github.com/apache/pulsar/pull/3772#issuecomment-470329785 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] codelipenghui commented on issue #3766: Support passing schema definition for JSON and AVRO schemas
codelipenghui commented on issue #3766: Support passing schema definition for JSON and AVRO schemas URL: https://github.com/apache/pulsar/pull/3766#issuecomment-470329631 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] rdhabalia commented on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager…
rdhabalia commented on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager… URL: https://github.com/apache/pulsar/pull/3760#issuecomment-470328994 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] srkukarni commented on issue #3770: Allow users to update everything in inputspecs except for isregexpattern
srkukarni commented on issue #3770: Allow users to update everything in inputspecs except for isregexpattern URL: https://github.com/apache/pulsar/pull/3770#issuecomment-470326835 updating input topic has the associated overhead of cleaning up subscriptions and such. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #3535: Deduplication causes a lot of Full GCs
merlimat commented on issue #3535: Deduplication causes a lot of Full GCs URL: https://github.com/apache/pulsar/issues/3535#issuecomment-470325628 As @hrsakai pointed out, the fix was ineffective because applied on a code path that's not being used. The problem is that that while the cursor is set as "inactive" in the beginning, a periodic check is flipping back the state to "active": https://github.com/apache/pulsar/blob/43380523c5269c152f61b2aa8f7b70281c770d1d/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L878-L885 Working on a fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai opened a new issue #3773: Update documentation for topic partitions auto update
jiazhai opened a new issue #3773: Update documentation for topic partitions auto update URL: https://github.com/apache/pulsar/issues/3773 Update documentation for topic partitions auto update This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated: On shutdown, kill the function instance thread after interrupt attempt (#3749)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 69faeaf On shutdown, kill the function instance thread after interrupt attempt (#3749) 69faeaf is described below commit 69faeaf80a70947fca18e36d47cdf7540560d637 Author: Matteo Merli AuthorDate: Wed Mar 6 14:39:26 2019 -0800 On shutdown, kill the function instance thread after interrupt attempt (#3749) --- .../org/apache/pulsar/functions/runtime/ThreadRuntime.java | 14 +++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ThreadRuntime.java b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ThreadRuntime.java index ad1002c..93246f2 100644 --- a/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ThreadRuntime.java +++ b/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ThreadRuntime.java @@ -44,6 +44,8 @@ class ThreadRuntime implements Runtime { // The thread that invokes the function private Thread fnThread; +private static final int THREAD_SHUTDOWN_TIMEOUT_MILLIS = 10_000; + @Getter private InstanceConfig instanceConfig; private JavaInstanceRunnable javaInstanceRunnable; @@ -114,13 +116,19 @@ class ThreadRuntime implements Runtime { } } +@SuppressWarnings("deprecation") @Override public void stop() { if (fnThread != null) { // interrupt the instance thread fnThread.interrupt(); try { -fnThread.join(); +// If the instance thread doesn't respond within some time, attempt to +// kill the thread +fnThread.join(THREAD_SHUTDOWN_TIMEOUT_MILLIS, 0); +if (fnThread.isAlive()) { +fnThread.stop(); +} } catch (InterruptedException e) { // ignore this } @@ -152,8 +160,8 @@ class ThreadRuntime implements Runtime { public CompletableFuture getAndResetMetrics() { return CompletableFuture.completedFuture(javaInstanceRunnable.getAndResetMetrics()); } - - + + @Override public CompletableFuture getMetrics(int instanceId) { return CompletableFuture.completedFuture(javaInstanceRunnable.getMetrics());
[GitHub] [pulsar] merlimat merged pull request #3749: On shutdown, kill the function instance thread after interrupt attempt
merlimat merged pull request #3749: On shutdown, kill the function instance thread after interrupt attempt URL: https://github.com/apache/pulsar/pull/3749 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] lovelle commented on issue #3748: [cpp client] Bugfix prevent dup consumer for same topic subscription
lovelle commented on issue #3748: [cpp client] Bugfix prevent dup consumer for same topic subscription URL: https://github.com/apache/pulsar/pull/3748#issuecomment-470306210 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] lovelle commented on issue #3746: [java client] Bugfix prevent dup consumers for same topic subscription
lovelle commented on issue #3746: [java client] Bugfix prevent dup consumers for same topic subscription URL: https://github.com/apache/pulsar/pull/3746#issuecomment-470306023 run java8 tests run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] lovelle removed a comment on issue #3748: [cpp client] Bugfix prevent dup consumer for same topic subscription
lovelle removed a comment on issue #3748: [cpp client] Bugfix prevent dup consumer for same topic subscription URL: https://github.com/apache/pulsar/pull/3748#issuecomment-469436084 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] lovelle removed a comment on issue #3746: [java client] Bugfix prevent dup consumers for same topic subscription
lovelle removed a comment on issue #3746: [java client] Bugfix prevent dup consumers for same topic subscription URL: https://github.com/apache/pulsar/pull/3746#issuecomment-470261127 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] lovelle removed a comment on issue #3748: [cpp client] Bugfix prevent dup consumer for same topic subscription
lovelle removed a comment on issue #3748: [cpp client] Bugfix prevent dup consumer for same topic subscription URL: https://github.com/apache/pulsar/pull/3748#issuecomment-470261008 run java8 tests run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] grantwwu edited a comment on issue #3715: Pulsar functions worker fails to initialize
grantwwu edited a comment on issue #3715: Pulsar functions worker fails to initialize URL: https://github.com/apache/pulsar/issues/3715#issuecomment-470291412 In broker logs. Running in a 3 node cluster. Unfortunately I'm not sure we can reproduce this anymore - it was breaking clusters frequently enough (once every 1 or 2 days?, with no obvious path to recovery) that we've downgraded to 2.2.1 entirely. I'm going to test out 2.3.1 when it comes out - we were also hitting some Jetty thread pool issues that merlimat said were getting fixed then. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] grantwwu commented on issue #3715: Pulsar functions worker fails to initialize
grantwwu commented on issue #3715: Pulsar functions worker fails to initialize URL: https://github.com/apache/pulsar/issues/3715#issuecomment-470291412 In broker logs. Running in a 3 node cluster. Unfortunately I'm not sure we can reproduce this anymore - it was breaking clusters frequently enough (once every 1 or 2 days?, with no obvious path to recovery) that we've downgraded to 2.2.1 entirely. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #3770: Allow users to update everything in inputspecs except for isregexpattern
jerrypeng commented on issue #3770: Allow users to update everything in inputspecs except for isregexpattern URL: https://github.com/apache/pulsar/pull/3770#issuecomment-470287776 @srkukarni but then why don't we allow users to update the actual input topic? Since serde and schema are kind of tied to the topic. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] srkukarni commented on issue #3717: Expose Secrets in Source/SinkContext
srkukarni commented on issue #3717: Expose Secrets in Source/SinkContext URL: https://github.com/apache/pulsar/pull/3717#issuecomment-470284433 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #3715: Pulsar functions worker fails to initialize
jerrypeng commented on issue #3715: Pulsar functions worker fails to initialize URL: https://github.com/apache/pulsar/issues/3715#issuecomment-470276983 @grantwwu how often have you seen this happen? Did you find the exceptions you listed above in function logs or worker/broker logs? How are you running cluster? Standalone? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] lovelle commented on issue #3746: [java client] Bugfix prevent dup consumers for same topic subscription
lovelle commented on issue #3746: [java client] Bugfix prevent dup consumers for same topic subscription URL: https://github.com/apache/pulsar/pull/3746#issuecomment-470273529 @sijie @jiazhai Great! :+1: thanks you both for the feedback. So when checks are success this pr will be ready to be merged! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] aahmed-se opened a new pull request #3772: Force before and after methods to always run
aahmed-se opened a new pull request #3772: Force before and after methods to always run URL: https://github.com/apache/pulsar/pull/3772 This should improve test stability This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] srkukarni commented on issue #3717: Expose Secrets in Source/SinkContext
srkukarni commented on issue #3717: Expose Secrets in Source/SinkContext URL: https://github.com/apache/pulsar/pull/3717#issuecomment-470271049 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager…
merlimat commented on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager… URL: https://github.com/apache/pulsar/pull/3760#issuecomment-470269568 run java8 tests run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] srkukarni commented on issue #3770: Allow users to update everything in inputspecs except for isregexpattern
srkukarni commented on issue #3770: Allow users to update everything in inputspecs except for isregexpattern URL: https://github.com/apache/pulsar/pull/3770#issuecomment-470262744 @jerrypeng I don't see a reason not to? Afterall one can experiment with different libraries of serialization changes to test which one works better. And also we do make sure that the types etc match. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] srkukarni commented on a change in pull request #3770: Allow users to update everything in inputspecs except for isregexpattern
srkukarni commented on a change in pull request #3770: Allow users to update everything in inputspecs except for isregexpattern URL: https://github.com/apache/pulsar/pull/3770#discussion_r263119829 ## File path: pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/SinkConfigUtils.java ## @@ -416,7 +416,7 @@ public static SinkConfig validateUpdate(SinkConfig existingConfig, SinkConfig ne if (!existingConfig.getInputSpecs().containsKey(topicName)) { throw new IllegalArgumentException("Input Topics cannot be altered"); } -if (!consumerConfig.equals(existingConfig.getInputSpecs().get(topicName))) { +if (consumerConfig.isRegexPattern() != existingConfig.getInputSpecs().get(topicName).isRegexPattern()) { throw new IllegalArgumentException("Input Specs mismatch"); Review comment: Changed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #3764: Fixed Reader.HasNext() in Go client
merlimat commented on a change in pull request #3764: Fixed Reader.HasNext() in Go client URL: https://github.com/apache/pulsar/pull/3764#discussion_r263119822 ## File path: pulsar-client-go/pulsar/reader_test.go ## @@ -218,3 +219,49 @@ func TestReaderCompaction(t *testing.T) { assert.Nil(t, msg) assert.NotNil(t, err) } + +func TestReaderHasNext(t *testing.T) { + topic := fmt.Sprintf("TestReaderHasNext-%d", rand.Int()) + ctx := context.Background() + + client, err := NewClient(ClientOptions{ + URL: "pulsar://localhost:6650", + //Logger: func(level logutil.LoggerLevel, file string, line int, message string) { Review comment: Good point.. :) removing these This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] srkukarni commented on a change in pull request #3770: Allow users to update everything in inputspecs except for isregexpattern
srkukarni commented on a change in pull request #3770: Allow users to update everything in inputspecs except for isregexpattern URL: https://github.com/apache/pulsar/pull/3770#discussion_r263119861 ## File path: pulsar-functions/utils/src/test/java/org/apache/pulsar/functions/utils/FunctionConfigUtilsTest.java ## @@ -157,6 +157,15 @@ public void testMergeDifferentInputs() { FunctionConfigUtils.validateUpdate(functionConfig, newFunctionConfig); } +@Test(expectedExceptions = IllegalArgumentException.class, expectedExceptionsMessageRegExp = "Input Specs mismatch") Review comment: added This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] lovelle commented on issue #3746: [java client] Bugfix prevent dup consumers for same topic subscription
lovelle commented on issue #3746: [java client] Bugfix prevent dup consumers for same topic subscription URL: https://github.com/apache/pulsar/pull/3746#issuecomment-470261127 run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] lovelle commented on issue #3748: [cpp client] Bugfix prevent dup consumer for same topic subscription
lovelle commented on issue #3748: [cpp client] Bugfix prevent dup consumer for same topic subscription URL: https://github.com/apache/pulsar/pull/3748#issuecomment-470261008 run java8 tests run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #3749: On shutdown, kill the function instance thread after interrupt attempt
jerrypeng commented on issue #3749: On shutdown, kill the function instance thread after interrupt attempt URL: https://github.com/apache/pulsar/pull/3749#issuecomment-470257979 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on issue #3770: Allow users to update everything in inputspecs except for isregexpattern
jerrypeng commented on issue #3770: Allow users to update everything in inputspecs except for isregexpattern URL: https://github.com/apache/pulsar/pull/3770#issuecomment-470257404 @srkukarni a general question: We should definitely allow users to update receiverQueueSize config in ConsumerConfig but does it make sense for users to also be able to update schemaType and serdeClassName ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on a change in pull request #3770: Allow users to update everything in inputspecs except for isregexpattern
jerrypeng commented on a change in pull request #3770: Allow users to update everything in inputspecs except for isregexpattern URL: https://github.com/apache/pulsar/pull/3770#discussion_r263113441 ## File path: pulsar-functions/utils/src/test/java/org/apache/pulsar/functions/utils/FunctionConfigUtilsTest.java ## @@ -157,6 +157,15 @@ public void testMergeDifferentInputs() { FunctionConfigUtils.validateUpdate(functionConfig, newFunctionConfig); } +@Test(expectedExceptions = IllegalArgumentException.class, expectedExceptionsMessageRegExp = "Input Specs mismatch") Review comment: Can we also add a test case that tests that other configs e.g. receiverQueueSize can be passed/updated without error This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng commented on a change in pull request #3770: Allow users to update everything in inputspecs except for isregexpattern
jerrypeng commented on a change in pull request #3770: Allow users to update everything in inputspecs except for isregexpattern URL: https://github.com/apache/pulsar/pull/3770#discussion_r263112143 ## File path: pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/SinkConfigUtils.java ## @@ -416,7 +416,7 @@ public static SinkConfig validateUpdate(SinkConfig existingConfig, SinkConfig ne if (!existingConfig.getInputSpecs().containsKey(topicName)) { throw new IllegalArgumentException("Input Topics cannot be altered"); } -if (!consumerConfig.equals(existingConfig.getInputSpecs().get(topicName))) { +if (consumerConfig.isRegexPattern() != existingConfig.getInputSpecs().get(topicName).isRegexPattern()) { throw new IllegalArgumentException("Input Specs mismatch"); Review comment: Can we modify the message of the exception to be more specific? That would help a lot in debugging This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #3768: [pulsar-function] intermittent test failure due to deadlock
merlimat commented on issue #3768: [pulsar-function] intermittent test failure due to deadlock URL: https://github.com/apache/pulsar/issues/3768#issuecomment-470254736 I think he's referring to the `PulsarWorkerAssignmentTest` test getting stuck during shutdown. There's already #3749 to fix that...though the ehm challenge is to get all these flaky to pass at once. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jerrypeng opened a new pull request #3771: fix: function config cleanupSubscription update bug
jerrypeng opened a new pull request #3771: fix: function config cleanupSubscription update bug URL: https://github.com/apache/pulsar/pull/3771 ### Motivation The are currently two bugs involving the config cleanupSubscription and updating a function: 1. when a user updates a function without changing the cleanupSubscription config, the cleanupSubscription config gets reverted to the default i.e. true 2. When a user attempts to update the cleanupSubscription config in a function, the config doesn't get changed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] srkukarni commented on issue #3768: [pulsar-function] intermittent test failure due to deadlock
srkukarni commented on issue #3768: [pulsar-function] intermittent test failure due to deadlock URL: https://github.com/apache/pulsar/issues/3768#issuecomment-470253042 @rdhabalia Could you please add more color on what the dead lock is? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch retry_creation updated (bbfb1cb -> 024831f)
This is an automated email from the ASF dual-hosted git repository. sanjeevrk pushed a change to branch retry_creation in repository https://gitbox.apache.org/repos/asf/pulsar.git. from bbfb1cb Added header new ccfb949 When the loadmanager leader is not available, fall through regular least loaded selection (#3688) new 0142247 Revert "Remove broker mode to handle persistent/non-persistent topics separately (#3348)" (#3713) new 7c7c3fb Use python 3.5 runtime and pulsar client (#3723) new bb9a4fb fix pulsar_message_set_replication_clusters (#3729) new 40aecc1 Fix Test localhost resolution errors (#3730) new bc8b380 Fail the source record if the write fails (#3706) new 5fd5730 [logging] make root log level configurable (#3661) new 65daec1 fix Jenkins pulsar-website-build error 571 (#3691) new 548c726 [Issue #3436][pulsar-broker] Creating REST Endpoint for non-partitioned topic creation (#3625) new 37a2fda [Issue 3275][pulsar-io]Support source and sink of flume (#3597) new d84b687 Update README.md with IDE setup for lombok (#3711) new e06a894 add cli client consumer for standalone (#3697) new d847c35 Exposing getSchemaVersion in the client by making it public. (#3744) new ffa2a68 Avoid debug noise by consumer ack-tracker (#2953) new 9dc3df6 Adjust the serving threads to have a minimum of threads (#3698) new 07cebb1 On publish failures, log error and count them as sys exceptions (#3704) new 33f1c55 fix function termination cleanup (#3751) new c39e7d1 Broker considers fail-over consumer priority-level (#2954) new 77332b0 [docs] Fixed typo remove-backlog-quotas to remove-backlog-quota in docs (#3757) new a662757 Multi version generic schema provider (#3756) new 024831f Merge branch 'master' into retry_creation The 2979 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: README.md | 28 ++ bin/pulsar | 1 + conf/broker.conf | 8 +- conf/log4j2.yaml | 4 +- conf/proxy.conf| 2 +- conf/standalone.conf | 9 + deployment/terraform-ansible/templates/broker.conf | 6 + docker/pulsar/Dockerfile | 8 +- docker/pulsar/pom.xml | 6 +- ...ar-client-37.sh => install-pulsar-client-35.sh} | 2 +- .../apache/pulsar/broker/ServiceConfiguration.java | 14 +- .../org/apache/pulsar/broker/PulsarService.java| 15 +- .../broker/admin/impl/PersistentTopicsBase.java| 12 + .../pulsar/broker/admin/v2/PersistentTopics.java | 16 + .../broker/loadbalance/LeaderElectionService.java | 17 + .../broker/loadbalance/impl/LoadManagerShared.java | 34 +- .../loadbalance/impl/ModularLoadManagerImpl.java | 43 +- .../loadbalance/impl/SimpleLoadManagerImpl.java| 27 +- .../pulsar/broker/namespace/NamespaceService.java | 11 +- .../AbstractDispatcherSingleActiveConsumer.java| 29 +- .../pulsar/broker/service/BrokerService.java | 16 + .../pulsar/broker/admin/AdminApiTlsAuthTest.java | 28 +- .../pulsar/broker/admin/PersistentTopicsTest.java | 5 +- .../broker/auth/MockedPulsarServiceBaseTest.java | 16 +- .../broker/loadbalance/LoadBalancerTest.java | 70 ++- .../loadbalance/ModularLoadManagerImplTest.java| 84 ++-- .../broker/service/ReplicatorGlobalNSTest.java | 15 +- .../pulsar/broker/service/ReplicatorTestBase.java | 4 +- .../broker/service/v1/V1_ReplicatorTest.java | 6 +- .../pulsar/client/api/NonPersistentTopicTest.java | 151 ++ .../client/api/SimpleProducerConsumerTest.java | 93 .../pulsar/client/impl/TopicFromMessageTest.java | 5 - .../worker/PulsarWorkerAssignmentTest.java | 78 ++- .../apache/pulsar/io/PulsarFunctionAdminTest.java | 2 +- .../apache/pulsar/io/PulsarFunctionE2ETest.java| 13 +- .../stats/client/PulsarBrokerStatsClientTest.java | 22 +- .../org/apache/pulsar/client/admin/Topics.java | 19 + .../pulsar/client/admin/internal/TopicsImpl.java | 19 + .../apache/pulsar/client/api/ConsumerBuilder.java | 20 + .../java/org/apache/pulsar/client/api/Message.java | 7 + pulsar-client-cpp/include/pulsar/c/message.h | 4 +- pulsar-client-cpp/lib/c/c_Message.cc | 8 +- pulsar-client-go/pulsar/c_message.go | 2 +- .../pulsar/admin/cli/PulsarAdminToolTest.java | 3 + .../org/apache/pulsar/admin/cli/CmdTopics.java | 14 + .../client/impl/BinaryProtoLookupService.java | 11 +- .../pulsar/client/impl/HttpLookupService.java | 18 +- ...
[GitHub] [pulsar] srkukarni opened a new pull request #3770: Allow users to update everything in inputspecs except for isregexpattern
srkukarni opened a new pull request #3770: Allow users to update everything in inputspecs except for isregexpattern URL: https://github.com/apache/pulsar/pull/3770 ### Motivation When doing function update, currently our update semantics are pretty restrictive. For example we cannot update any property in the consumerConfig in the inputSpecs. However apart from isRegexPattern, pretty much all parameters can be updated. This pr modifies the update behavior to do these kind of updates ### Modifications *Describe the modifications you've done.* ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API: (yes / no) - The schema: (yes / no / don't know) - The default values of configurations: (yes / no) - The wire protocol: (yes / no) - The rest endpoints: (yes / no) - The admin cli options: (yes / no) - Anything that affects deployment: (yes / no / don't know) ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a followup issue for adding the documentation This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #3750: Negative acks in C++ client
merlimat commented on a change in pull request #3750: Negative acks in C++ client URL: https://github.com/apache/pulsar/pull/3750#discussion_r263103180 ## File path: pulsar-client-cpp/tests/BasicEndToEndTest.cc ## @@ -99,7 +100,7 @@ static void sendCallBackWithDelay(Result r, const Message& msg, std::string pref class EncKeyReader : public CryptoKeyReader { private: -void readFile(std::string fileName, std::string& fileContents) const { +void readFile(std::string fileName, std::string &fileContents) const { Review comment: This is existing code and it was just one space reformatted by `make format`. In any case the functional change should be done in a different PR. Given this is for test, I'm not particular concerned about a string copy when loading the key file. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] grantwwu opened a new issue #3769: Websocket interface does not decode URLencoding
grantwwu opened a new issue #3769: Websocket interface does not decode URLencoding URL: https://github.com/apache/pulsar/issues/3769 **Describe the bug** The Pulsar websocket server does not appear to decode URL encoded URLs properly. This leads to consuming from (and presuming producing to, but I didn't test this) the wrong topic. **To Reproduce** Steps to reproduce the behavior: 1. Attempt to consume from a *new* topic whose name includes an RFC 3986 reserved character. For this bug report, I am using the topic name `public/default/pusheennstormy[]`. To encode this into a valid URL, note that you will need to URL encode the square brackets. The resulting URL ought to look something like this: `pulsar-broker.petuum-system:8080/ws/v2/consumer/persistent/public/default/pusheennstormy%5B%5D/3ab32f35-6c3f-4e8a-a120-409091bb3cea` 2. Note that the the new topic created is not `public/default/pusheennstormy[]` as expected, but actually is the URL encoded form. **Expected behavior** The `public/default/pusheennstormy[]` topic should be created. **Log lines** ``` 19:14:58.299 [pulsar-client-io-53-6] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/pusheennstormy%5B%5D][3ab32f35-6c3f-4e8a-a120-409091bb3cea] Subscribing to topic on cnx [id: 0x190c12dc, L:/10.244.1.53:37080 - R:10.244.2.32/10.244.2.32:6650] 19:14:58.358 [pulsar-client-io-53-6] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/default/pusheennstormy%5B%5D][3ab32f35-6c3f-4e8a-a120-409091bb3cea] Subscribed to topic on 10.244.2.32/10.244.2.32:6650 -- consumer: 3 19:14:58.359 [pulsar-web-30-25] INFO org.eclipse.jetty.server.RequestLog - 10.244.0.61 - - [06/Mar/2019:19:14:58 +] "GET //pulsar-broker.petuum-system:8080/ws/v2/consumer/persistent/public/default/pusheennstormy%5B%5D/3ab32f35-6c3f-4e8a-a120-409091bb3cea HTTP/1.1" 101 0 "-" "-" 70 19:14:58.359 [pulsar-web-30-25] INFO org.apache.pulsar.websocket.AbstractWebSocketHandler - [/10.244.0.61:46224] New WebSocket session on topic persistent://public/default/pusheennstormy%5B%5D ``` **Desktop (please complete the following information):** - Running on Pulsar 2.2.1 using the `pulsar-all` docker image w/ Kubernetes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jai1 commented on issue #3750: Negative acks in C++ client
jai1 commented on issue #3750: Negative acks in C++ client URL: https://github.com/apache/pulsar/pull/3750#issuecomment-470243091 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jai1 commented on a change in pull request #3750: Negative acks in C++ client
jai1 commented on a change in pull request #3750: Negative acks in C++ client URL: https://github.com/apache/pulsar/pull/3750#discussion_r263097629 ## File path: pulsar-client-cpp/tests/BasicEndToEndTest.cc ## @@ -99,7 +100,7 @@ static void sendCallBackWithDelay(Result r, const Message& msg, std::string pref class EncKeyReader : public CryptoKeyReader { private: -void readFile(std::string fileName, std::string& fileContents) const { +void readFile(std::string fileName, std::string &fileContents) const { Review comment: `const std::string& fileName` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] srkukarni commented on a change in pull request #3717: Expose Secrets in Source/SinkContext
srkukarni commented on a change in pull request #3717: Expose Secrets in Source/SinkContext URL: https://github.com/apache/pulsar/pull/3717#discussion_r263097424 ## File path: pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/SchedulerManager.java ## @@ -105,25 +105,12 @@ public SchedulerManager(WorkerConfig workerConfig, PulsarClient pulsarClient, Pu .compressionType(CompressionType.LZ4) .sendTimeout(0, TimeUnit.MILLISECONDS) .createAsync().get(10, TimeUnit.SECONDS); +} catch (Exception e) { Review comment: Oops, looks like bad commit. Reverted This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jai1 commented on issue #3761: Updated consumer example to use negative acks
jai1 commented on issue #3761: Updated consumer example to use negative acks URL: https://github.com/apache/pulsar/pull/3761#issuecomment-470240572 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jai1 commented on a change in pull request #3633: [pulsar-broker] Fix deadlock: add zk-operation timeout for blocking call on zk-cache
jai1 commented on a change in pull request #3633: [pulsar-broker] Fix deadlock: add zk-operation timeout for blocking call on zk-cache URL: https://github.com/apache/pulsar/pull/3633#discussion_r263094693 ## File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/web/PulsarWebResource.java ## @@ -609,9 +608,10 @@ protected void validateTopicOwnership(TopicName topicName, boolean authoritative * @throws Exception */ protected void validateGlobalNamespaceOwnership(NamespaceName namespace) { +int timeout = pulsar().getConfiguration().getZooKeeperOperationTimeoutSeconds(); Review comment: nit: var name -> timeoutInSecs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jai1 commented on a change in pull request #3633: [pulsar-broker] Fix deadlock: add zk-operation timeout for blocking call on zk-cache
jai1 commented on a change in pull request #3633: [pulsar-broker] Fix deadlock: add zk-operation timeout for blocking call on zk-cache URL: https://github.com/apache/pulsar/pull/3633#discussion_r263094012 ## File path: conf/broker.conf ## @@ -61,6 +61,9 @@ failureDomainsEnabled=false # Zookeeper session timeout in milliseconds zooKeeperSessionTimeoutMillis=3 +# ZooKeeper operation timeout in seconds Review comment: nit: Other config around this are in Ms so I would suggest to keep the TimeUnit in millis. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jai1 commented on issue #3633: [pulsar-broker] Fix deadlock: add zk-operation timeout for blocking call on zk-cache
jai1 commented on issue #3633: [pulsar-broker] Fix deadlock: add zk-operation timeout for blocking call on zk-cache URL: https://github.com/apache/pulsar/pull/3633#issuecomment-470238478 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] rdhabalia commented on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager…
rdhabalia commented on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager… URL: https://github.com/apache/pulsar/pull/3760#issuecomment-470238447 created issue #3768 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] rdhabalia opened a new issue #3768: [pulsar-function] intermittent test failure due to deadlock
rdhabalia opened a new issue #3768: [pulsar-function] intermittent test failure due to deadlock URL: https://github.com/apache/pulsar/issues/3768 [org.apache.pulsar.functions.worker.PulsarWorkerAssignmentTest.shutdown()](https://builds.apache.org/job/pulsar_precommit_java8/7046/testReport/junit/org.apache.pulsar.functions.worker/PulsarWorkerAssignmentTest/shutdown/) has intermittent failure due to deadlock into function-worker. Disabling this for now and we have to enable back once deadlock issue is fixed at function-worker. ``` Error Message Method org.apache.pulsar.functions.worker.PulsarWorkerAssignmentTest.shutdown() didn't finish within the time-out 6 Stacktrace org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.functions.worker.PulsarWorkerAssignmentTest.shutdown() didn't finish within the time-out 6 at io.netty.channel.epoll.Native.eventFdWrite(Native Method) at io.netty.channel.epoll.EpollEventLoop.wakeup(EpollEventLoop.java:169) at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:799) at io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:1013) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:825) at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:837) at org.apache.pulsar.client.impl.ClientCnx.sendRequestWithId(ClientCnx.java:685) at org.apache.pulsar.client.impl.ProducerImpl.closeAsync(ProducerImpl.java:609) at org.apache.pulsar.functions.sink.PulsarSink$PulsarSinkProcessorBase.close(PulsarSink.java:130) at org.apache.pulsar.functions.sink.PulsarSink.close(PulsarSink.java:306) at org.apache.pulsar.functions.instance.JavaInstanceRunnable.close(JavaInstanceRunnable.java:482) at org.apache.pulsar.functions.runtime.ThreadRuntime.stop(ThreadRuntime.java:128) at org.apache.pulsar.functions.runtime.RuntimeSpawner.close(RuntimeSpawner.java:151) at org.apache.pulsar.functions.worker.FunctionActioner.stopFunction(FunctionActioner.java:243) at org.apache.pulsar.functions.worker.FunctionRuntimeManager.conditionallyStopFunction(FunctionRuntimeManager.java:868) at org.apache.pulsar.functions.worker.FunctionRuntimeManager.stopFunction(FunctionRuntimeManager.java:441) at org.apache.pulsar.functions.worker.FunctionRuntimeManager.lambda$stopAllOwnedFunctions$1(FunctionRuntimeManager.java:429) at org.apache.pulsar.functions.worker.FunctionRuntimeManager$$Lambda$795/1890737615.accept(Unknown Source) at java.util.HashMap$Values.forEach(HashMap.java:981) at org.apache.pulsar.functions.worker.FunctionRuntimeManager.stopAllOwnedFunctions(FunctionRuntimeManager.java:426) at org.apache.pulsar.functions.worker.FunctionRuntimeManager.close(FunctionRuntimeManager.java:813) at org.apache.pulsar.functions.worker.WorkerService.stop(WorkerService.java:218) at org.apache.pulsar.functions.worker.PulsarWorkerAssignmentTest.shutdown(PulsarWorkerAssignmentTest.java:135) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124) at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54) at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jai1 commented on a change in pull request #3764: Fixed Reader.HasNext() in Go client
jai1 commented on a change in pull request #3764: Fixed Reader.HasNext() in Go client URL: https://github.com/apache/pulsar/pull/3764#discussion_r263092680 ## File path: pulsar-client-go/pulsar/reader_test.go ## @@ -218,3 +219,49 @@ func TestReaderCompaction(t *testing.T) { assert.Nil(t, msg) assert.NotNil(t, err) } + +func TestReaderHasNext(t *testing.T) { + topic := fmt.Sprintf("TestReaderHasNext-%d", rand.Int()) + ctx := context.Background() + + client, err := NewClient(ClientOptions{ + URL: "pulsar://localhost:6650", + //Logger: func(level logutil.LoggerLevel, file string, line int, message string) { Review comment: Do you want to checkin with these comments This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] rdhabalia commented on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager…
rdhabalia commented on issue #3760: Fix intermittent test: BrokerServiceLookupTest.testModularLoadManager… URL: https://github.com/apache/pulsar/pull/3760#issuecomment-470237356 I am going to disable [org.apache.pulsar.functions.worker.PulsarWorkerAssignmentTest.shutdown()](https://builds.apache.org/job/pulsar_precommit_java8/7046/testReport/junit/org.apache.pulsar.functions.worker/PulsarWorkerAssignmentTest/shutdown/) test for now because it's blocking other PR. It's mainly happening due to deadlock in function-worker while stopping the function during shutdown. I will create separate issue to fix the deadlock and then we can enable this test again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #3764: Fixed Reader.HasNext() in Go client
merlimat commented on a change in pull request #3764: Fixed Reader.HasNext() in Go client URL: https://github.com/apache/pulsar/pull/3764#discussion_r263083466 ## File path: pulsar-client-cpp/lib/ConsumerImpl.cc ## @@ -1039,8 +1040,7 @@ void ConsumerImpl::hasMessageAvailableAsync(HasMessageAvailableCallback callback return; } -BrokerGetLastMessageIdCallback callback1 = [this, lastDequed, callback](Result result, - MessageId messageId) { +getLastMessageIdAsync([this, lastDequed, callback](Result result, MessageId messageId) { if (result == ResultOk) { Review comment: I don't think that this would improve readability This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jai1 commented on a change in pull request #3764: Fixed Reader.HasNext() in Go client
jai1 commented on a change in pull request #3764: Fixed Reader.HasNext() in Go client URL: https://github.com/apache/pulsar/pull/3764#discussion_r263080874 ## File path: pulsar-client-cpp/lib/ConsumerImpl.cc ## @@ -1039,8 +1040,7 @@ void ConsumerImpl::hasMessageAvailableAsync(HasMessageAvailableCallback callback return; } -BrokerGetLastMessageIdCallback callback1 = [this, lastDequed, callback](Result result, - MessageId messageId) { +getLastMessageIdAsync([this, lastDequed, callback](Result result, MessageId messageId) { if (result == ResultOk) { Review comment: nit: can you refactor it to ``` const auto ret = (result == ResultOk && messageId > lastDequed && messageId.entryId() != -1); callback(result, ret); ``` or even not use ret. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services