[GitHub] liketic commented on issue #3338: Remove global zookeeper option in code base
liketic commented on issue #3338: Remove global zookeeper option in code base URL: https://github.com/apache/pulsar/pull/3338#issuecomment-453716065 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] liketic commented on issue #3307: Remove global-zookeeper in documentation
liketic commented on issue #3307: Remove global-zookeeper in documentation URL: https://github.com/apache/pulsar/pull/3307#issuecomment-453715350 @sijie I updated this PR for removing from documentation, could you please review again? Thanks in advance. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat closed pull request #3360: Support TLS authentication and authorization in standalone mode
merlimat closed pull request #3360: Support TLS authentication and
authorization in standalone mode
URL: https://github.com/apache/pulsar/pull/3360
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
index 8711bd84d6..c6ee4dc762 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
@@ -272,8 +272,16 @@ void start() throws Exception {
workerConfig = WorkerConfig.load(this.getFnWorkerConfigFile());
}
// worker talks to local broker
-workerConfig.setPulsarServiceUrl("pulsar://127.0.0.1:" +
config.getBrokerServicePort().get());
-workerConfig.setPulsarWebServiceUrl("http://127.0.0.1:"; +
config.getWebServicePort().get());
+boolean useTls = workerConfig.isUseTls();
+String localhost = "127.0.0.1";
+String pulsarServiceUrl = useTls
+? PulsarService.brokerUrlTls(localhost,
config.getBrokerServicePortTls().get())
+: PulsarService.brokerUrl(localhost,
config.getBrokerServicePort().get());
+String webServiceUrl = useTls
+? PulsarService.webAddressTls(localhost,
config.getWebServicePortTls().get())
+: PulsarService.webAddress(localhost,
config.getWebServicePort().get());
+workerConfig.setPulsarServiceUrl(pulsarServiceUrl);
+workerConfig.setPulsarWebServiceUrl(webServiceUrl);
if (!this.isNoStreamStorage()) {
// only set the state storage service url when state is
enabled.
workerConfig.setStateStorageServiceUrl("bk://127.0.0.1:" +
this.getStreamStoragePort());
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[pulsar] branch master updated: Support TLS authentication and authorization in standalone mode (#3360)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 617712e Support TLS authentication and authorization in standalone
mode (#3360)
617712e is described below
commit 617712e2913c8f53873e60631905b7d14ccd947b
Author: tuteng
AuthorDate: Sat Jan 12 10:41:38 2019 +0800
Support TLS authentication and authorization in standalone mode (#3360)
---
.../src/main/java/org/apache/pulsar/PulsarStandalone.java| 12 ++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
index 8711bd8..c6ee4dc 100644
--- a/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
+++ b/pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandalone.java
@@ -272,8 +272,16 @@ public class PulsarStandalone implements AutoCloseable {
workerConfig = WorkerConfig.load(this.getFnWorkerConfigFile());
}
// worker talks to local broker
-workerConfig.setPulsarServiceUrl("pulsar://127.0.0.1:" +
config.getBrokerServicePort().get());
-workerConfig.setPulsarWebServiceUrl("http://127.0.0.1:"; +
config.getWebServicePort().get());
+boolean useTls = workerConfig.isUseTls();
+String localhost = "127.0.0.1";
+String pulsarServiceUrl = useTls
+? PulsarService.brokerUrlTls(localhost,
config.getBrokerServicePortTls().get())
+: PulsarService.brokerUrl(localhost,
config.getBrokerServicePort().get());
+String webServiceUrl = useTls
+? PulsarService.webAddressTls(localhost,
config.getWebServicePortTls().get())
+: PulsarService.webAddress(localhost,
config.getWebServicePort().get());
+workerConfig.setPulsarServiceUrl(pulsarServiceUrl);
+workerConfig.setPulsarWebServiceUrl(webServiceUrl);
if (!this.isNoStreamStorage()) {
// only set the state storage service url when state is
enabled.
workerConfig.setStateStorageServiceUrl("bk://127.0.0.1:" +
this.getStreamStoragePort());
[GitHub] merlimat closed pull request #3357: fix dep to make build compatible with jdk11
merlimat closed pull request #3357: fix dep to make build compatible with jdk11 URL: https://github.com/apache/pulsar/pull/3357 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/pulsar-sql/presto-distribution/pom.xml b/pulsar-sql/presto-distribution/pom.xml index 9f65697985..a99aea4336 100644 --- a/pulsar-sql/presto-distribution/pom.xml +++ b/pulsar-sql/presto-distribution/pom.xml @@ -60,6 +60,10 @@ org.openjdk.jol jol-core + +com.sun +tools + This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated: fix dep to make build compatible with jdk11 (#3357)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new 1484711 fix dep to make build compatible with jdk11 (#3357) 1484711 is described below commit 1484711d69fc25e50a8ff0a0aa7d6ee1180c1fd0 Author: Rajan Dhabalia AuthorDate: Fri Jan 11 18:24:53 2019 -0800 fix dep to make build compatible with jdk11 (#3357) --- pulsar-sql/presto-distribution/pom.xml | 4 1 file changed, 4 insertions(+) diff --git a/pulsar-sql/presto-distribution/pom.xml b/pulsar-sql/presto-distribution/pom.xml index 9f65697..a99aea4 100644 --- a/pulsar-sql/presto-distribution/pom.xml +++ b/pulsar-sql/presto-distribution/pom.xml @@ -60,6 +60,10 @@ org.openjdk.jol jol-core + +com.sun +tools +
[GitHub] lovelle edited a comment on issue #3337: Bug fixes/Improvement for notify pending receive method
lovelle edited a comment on issue #3337: Bug fixes/Improvement for notify pending receive method URL: https://github.com/apache/pulsar/pull/3337#issuecomment-453243731 @sijie @merlimat Could you please review this? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #3363: Enable stats to be recovered by Kubernetes runtime
srkukarni commented on issue #3363: Enable stats to be recovered by Kubernetes runtime URL: https://github.com/apache/pulsar/pull/3363#issuecomment-453694947 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni opened a new pull request #3363: Enable stats to be recovered by Kubernetes runtime
srkukarni opened a new pull request #3363: Enable stats to be recovered by Kubernetes runtime URL: https://github.com/apache/pulsar/pull/3363 ### Motivation The first implementation of KubernetesRuntime omitted implementing getstats from functions. This pr adds that capability ### Modifications Describe the modifications you've done. ### Result After your change, what will change. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni closed pull request #3336: Added HTTP Support
srkukarni closed pull request #3336: Added HTTP Support
URL: https://github.com/apache/pulsar/pull/3336
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySource.java
b/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySource.java
index 215bd344a6..1e799d8f58 100644
--- a/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySource.java
+++ b/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySource.java
@@ -18,20 +18,21 @@
*/
package org.apache.pulsar.io.netty;
+import java.util.Map;
+
import org.apache.pulsar.io.core.PushSource;
import org.apache.pulsar.io.core.SourceContext;
import org.apache.pulsar.io.core.annotations.Connector;
import org.apache.pulsar.io.core.annotations.IOType;
import org.apache.pulsar.io.netty.server.NettyServer;
-import java.util.Map;
/**
- * A simple Netty Tcp or Udp Source connector to listen Tcp/Udp messages and
write to user-defined Pulsar topic
+ * A simple Netty Source connector to listen for incoming messages and write
to user-defined Pulsar topic.
*/
@Connector(
name = "netty",
type = IOType.SOURCE,
-help = "A simple Netty Tcp or Udp Source connector to listen Tcp/Udp
messages and write to user-defined Pulsar topic",
+help = "A simple Netty Source connector to listen for incoming messages
and write to user-defined Pulsar topic",
configClass = NettySourceConfig.class)
public class NettySource extends PushSource {
diff --git
a/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySourceConfig.java
b/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySourceConfig.java
index f5d40e930a..1ef4c35a27 100644
---
a/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySourceConfig.java
+++
b/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySourceConfig.java
@@ -20,17 +20,22 @@
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
-import lombok.*;
-import lombok.experimental.Accessors;
-import org.apache.pulsar.io.core.annotations.FieldDoc;
import java.io.File;
import java.io.IOException;
import java.io.Serializable;
import java.util.Map;
+import lombok.Data;
+import lombok.EqualsAndHashCode;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+import lombok.experimental.Accessors;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
/**
- * Netty Tcp or Udp Source Connector Config.
+ * Netty Source Connector Config.
*/
@Data
@Setter
@@ -45,7 +50,7 @@
@FieldDoc(
required = true,
defaultValue = "tcp",
-help = "The tcp or udp network protocols")
+help = "The network protocol to use, supported values are 'tcp',
'udp', and 'http'")
private String type = "tcp";
@FieldDoc(
@@ -63,8 +68,8 @@
@FieldDoc(
required = true,
defaultValue = "1",
-help = "The number of threads of Netty Tcp Server to accept
incoming connections and " +
-"handle the traffic of the accepted connections")
+help = "The number of threads of Netty Tcp Server to accept
incoming connections and "
++ "handle the traffic of the accepted connections")
private int numberOfThreads = 1;
public static NettySourceConfig load(Map map) throws
IOException {
diff --git
a/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/http/NettyHttpChannelInitializer.java
b/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/http/NettyHttpChannelInitializer.java
new file mode 100644
index 00..dce26fbe6c
--- /dev/null
+++
b/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/http/NettyHttpChannelInitializer.java
@@ -0,0 +1,48 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.io.netty.http;
+
+import io.netty.channel.ChannelInboundHandlerAdapte
[pulsar] branch master updated: Added HTTP Support (#3336)
This is an automated email from the ASF dual-hosted git repository.
sanjeevrk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 7ec205a Added HTTP Support (#3336)
7ec205a is described below
commit 7ec205ad5733bbbc9d99da16ed4721221bb5b315
Author: David Kjerrumgaard <35466513+david-stream...@users.noreply.github.com>
AuthorDate: Fri Jan 11 13:27:36 2019 -0800
Added HTTP Support (#3336)
* Added HTTP Support
* Updated documentation and removed duplicate code
* Added unit test for NettyHttpChannelInitializer
---
.../org/apache/pulsar/io/netty/NettySource.java| 7 +-
.../apache/pulsar/io/netty/NettySourceConfig.java | 19 ++-
.../NettyHttpChannelInitializer.java} | 19 ++-
.../io/netty/http/NettyHttpServerHandler.java | 144 +
.../package-info.java} | 26 +---
...tyChannelInitializer.java => package-info.java} | 26 +---
.../io/netty/server/NettyChannelInitializer.java | 2 +-
.../apache/pulsar/io/netty/server/NettyServer.java | 70 ++
.../pulsar/io/netty/server/NettyServerHandler.java | 13 +-
...tyChannelInitializer.java => package-info.java} | 26 +---
.../http/NettyHttpChannelInitializerTest.java} | 32 ++---
site2/docs/io-connectors.md| 2 +-
site2/docs/io-netty.md | 4 +-
13 files changed, 250 insertions(+), 140 deletions(-)
diff --git
a/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySource.java
b/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySource.java
index 215bd34..1e799d8 100644
--- a/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySource.java
+++ b/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySource.java
@@ -18,20 +18,21 @@
*/
package org.apache.pulsar.io.netty;
+import java.util.Map;
+
import org.apache.pulsar.io.core.PushSource;
import org.apache.pulsar.io.core.SourceContext;
import org.apache.pulsar.io.core.annotations.Connector;
import org.apache.pulsar.io.core.annotations.IOType;
import org.apache.pulsar.io.netty.server.NettyServer;
-import java.util.Map;
/**
- * A simple Netty Tcp or Udp Source connector to listen Tcp/Udp messages and
write to user-defined Pulsar topic
+ * A simple Netty Source connector to listen for incoming messages and write
to user-defined Pulsar topic.
*/
@Connector(
name = "netty",
type = IOType.SOURCE,
-help = "A simple Netty Tcp or Udp Source connector to listen Tcp/Udp
messages and write to user-defined Pulsar topic",
+help = "A simple Netty Source connector to listen for incoming messages
and write to user-defined Pulsar topic",
configClass = NettySourceConfig.class)
public class NettySource extends PushSource {
diff --git
a/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySourceConfig.java
b/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySourceConfig.java
index f5d40e9..1ef4c35 100644
---
a/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySourceConfig.java
+++
b/pulsar-io/netty/src/main/java/org/apache/pulsar/io/netty/NettySourceConfig.java
@@ -20,17 +20,22 @@ package org.apache.pulsar.io.netty;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
-import lombok.*;
-import lombok.experimental.Accessors;
-import org.apache.pulsar.io.core.annotations.FieldDoc;
import java.io.File;
import java.io.IOException;
import java.io.Serializable;
import java.util.Map;
+import lombok.Data;
+import lombok.EqualsAndHashCode;
+import lombok.Getter;
+import lombok.Setter;
+import lombok.ToString;
+import lombok.experimental.Accessors;
+import org.apache.pulsar.io.core.annotations.FieldDoc;
+
/**
- * Netty Tcp or Udp Source Connector Config.
+ * Netty Source Connector Config.
*/
@Data
@Setter
@@ -45,7 +50,7 @@ public class NettySourceConfig implements Serializable {
@FieldDoc(
required = true,
defaultValue = "tcp",
-help = "The tcp or udp network protocols")
+help = "The network protocol to use, supported values are 'tcp',
'udp', and 'http'")
private String type = "tcp";
@FieldDoc(
@@ -63,8 +68,8 @@ public class NettySourceConfig implements Serializable {
@FieldDoc(
required = true,
defaultValue = "1",
-help = "The number of threads of Netty Tcp Server to accept
incoming connections and " +
-"handle the traffic of the accepted connections")
+help = "The number of threads of Netty Tcp Server to accept
incoming connections and "
++ "handle the traffic of the accepted connections")
private int numberOfThreads = 1;
public static NettySourceConfig load(Map map) throws
IOException
[GitHub] sautran commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d
sautran commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d URL: https://github.com/apache/pulsar/issues/3362#issuecomment-453661576 remove data and logs directory, create a brand new topic. before starting consumer, server is good. as soon as start the consumer, no producer. it has same error. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sautran removed a comment on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d
sautran removed a comment on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d URL: https://github.com/apache/pulsar/issues/3362#issuecomment-453655217 i am running local standalone mode. I already removed the data and logs directory, server still kept throwing this error. What files/directory do I need to delete to do the fresh restart.? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #3324: Add Windowfunction interface to functions api
srkukarni commented on issue #3324: Add Windowfunction interface to functions api URL: https://github.com/apache/pulsar/pull/3324#issuecomment-453655674 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sautran commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d
sautran commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d URL: https://github.com/apache/pulsar/issues/3362#issuecomment-453655217 i am running local standalone mode. I already removed the data and logs directory, server still kept throwing this error. What files/directory do I need to delete to do the fresh restart.? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #3338: Remove global zookeeper option in code base
srkukarni commented on issue #3338: Remove global zookeeper option in code base URL: https://github.com/apache/pulsar/pull/3338#issuecomment-453645118 rerun java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on issue #3321: Add doc entry for packaging python dependencies.
srkukarni commented on issue #3321: Add doc entry for packaging python dependencies. URL: https://github.com/apache/pulsar/pull/3321#issuecomment-453643763 rerun integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] srkukarni commented on a change in pull request #3324: Add Windowfunction interface to functions api
srkukarni commented on a change in pull request #3324: Add Windowfunction
interface to functions api
URL: https://github.com/apache/pulsar/pull/3324#discussion_r247239113
##
File path:
pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/WindowFunction.java
##
@@ -0,0 +1,34 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.functions.api;
+
+import java.util.Collection;
+
+/**
+ * This is the interface of the windowed function api. The process method is
called
+ * for every triggered window.
+ */
+@FunctionalInterface
+public interface WindowFunction {
+/**
+ * Process the input.
+ * @return the output
+ */
+O process(Collection input, WindowContext context) throws Exception;
Review comment:
@jerrypeng as discussed, I've updated the interface to reflect our
discussions. Please take a look again. Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] asfgit commented on issue #3357: fix dep to make build compatible with jdk11
asfgit commented on issue #3357: fix dep to make build compatible with jdk11 URL: https://github.com/apache/pulsar/pull/3357#issuecomment-453637891 SUCCESS --none-- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sautran commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d
sautran commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d URL: https://github.com/apache/pulsar/issues/3362#issuecomment-453636095 Consumer log: 20190111.142908.328 INFO [org.apache.kafka.clients.consumer.ConsumerConfig] ConsumerConfig values: auto.commit.interval.ms = 5000 auto.offset.reset = latest bootstrap.servers = [pulsar://localhost:6650] check.crcs = true client.id = connections.max.idle.ms = 54 enable.auto.commit = false exclude.internal.topics = true fetch.max.bytes = 52428800 fetch.max.wait.ms = 500 fetch.min.bytes = 1 group.id = 222 heartbeat.interval.ms = 3000 interceptor.classes = null key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer max.partition.fetch.bytes = 1048576 max.poll.interval.ms = 30 max.poll.records = 500 metadata.max.age.ms = 30 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 3 partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor] receive.buffer.bytes = 65536 reconnect.backoff.ms = 50 request.timeout.ms = 305000 retry.backoff.ms = 100 sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 6 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT send.buffer.bytes = 131072 session.timeout.ms = 1 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer - 20190111.142908.474 WARN [org.apache.kafka.clients.consumer.ConsumerConfig] The configuration 'AAA' was supplied but isn't a known config. - 20190111.142908.475 WARN [org.apache.kafka.clients.consumer.ConsumerConfig] The configuration 'BBB' was supplied but isn't a known config. - 20190111.142908.475 WARN [org.apache.kafka.clients.consumer.ConsumerConfig] The configuration 'thread_number' was supplied but isn't a known config. - 20190111.142908.475 WARN [org.apache.kafka.clients.consumer.ConsumerConfig] The configuration 'socket.timeout.ms' was supplied but isn't a known config. - 20190111.142908.477 INFO [org.apache.kafka.common.utils.AppInfoParser] Kafka version : 0.10.2.1 - 20190111.142908.477 INFO [org.apache.kafka.common.utils.AppInfoParser] Kafka commitId : e89bffd6b2eff799 - 20190111.143140.410 INFO [org.apache.kafka.clients.consumer.ConsumerConfig] ConsumerConfig values: auto.commit.interval.ms = 5000 auto.offset.reset = latest bootstrap.servers = [pulsar://localhost:6650] check.crcs = true client.id = connections.max.idle.ms = 54 enable.auto.commit = false exclude.internal.topics = true fetch.max.bytes = 52428800 fetch.max.wait.ms = 500 fetch.min.bytes = 1 group.id = 222 heartbeat.interval.ms = 3000 interceptor.classes = null key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer max.partition.fetch.bytes = 1048576 max.poll.interval.ms = 30 max.poll.records = 500 metadata.max.age.ms = 30 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 3 partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor] receive.buffer.bytes = 65536 reconnect.backoff.ms = 50 request.timeout.ms = 305000 retry.backoff.ms = 100 sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 6 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT send.buffer.bytes = 131072 session.timeout.ms = 1 ssl.cipher.suites = null
[GitHub] merlimat commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d
merlimat commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d URL: https://github.com/apache/pulsar/issues/3362#issuecomment-453634289 Ok, that's strange. Can you also include the client logs? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sautran commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d
sautran commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d URL: https://github.com/apache/pulsar/issues/3362#issuecomment-453633946 I set bootstrap.servers pulsar://localhost:6650 in the properties This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d
merlimat commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d URL: https://github.com/apache/pulsar/issues/3362#issuecomment-453633054 This seems like it's trying to connect the Pulsar service using HTTP? What service url did you specify? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sautran commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d
sautran commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d URL: https://github.com/apache/pulsar/issues/3362#issuecomment-453631964 Ok. I set the consumer props.put(KafkaKeys.VALUE_DESERIALIZER, ByteArrayDeserializer.class.getName()); props.put(KafkaKeys.KEY_DESERIALIZER, StringDeserializer.class.getName()); in producer, I have the same setting but it is the serializer. now I am getting the following error in the pulsar server: 14:16:27.791 [pulsar-io-49-12] WARN org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:51545] Got exception IndexOutOfBoundsException : writerIndex: 1179652 (expected: readerIndex(4) <= writerIndex <= capacity(20)) java.lang.IndexOutOfBoundsException: writerIndex: 1179652 (expected: readerIndex(4) <= writerIndex <= capacity(20)) at io.netty.buffer.AbstractByteBuf.writerIndex(AbstractByteBuf.java:118) ~[io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at org.apache.pulsar.common.api.PulsarDecoder.channelRead(PulsarDecoder.java:83) ~[org.apache.pulsar-pulsar-common-2.2.1.jar:2.2.1] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:146) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:649) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:584) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:501) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.22.Final.jar:4.1.22.Final] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45] This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d
merlimat commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d URL: https://github.com/apache/pulsar/issues/3362#issuecomment-453619428 This error is coming from consumer, so it should be the `StringDeserializer` there This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sautran commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d
sautran commented on issue #3362: java.lang.IllegalArgumentException: Illegal
base64 character 2d
URL: https://github.com/apache/pulsar/issues/3362#issuecomment-453617931
@merlimat , yes, I set the key as StringSerializer
props.put("key.serializer",
StringSerializer.class.getName());
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] merlimat commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d
merlimat commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d URL: https://github.com/apache/pulsar/issues/3362#issuecomment-453615985 (Above code was taken from https://github.com/apache/pulsar/blob/ef82f045257ae839c0208c75b1b31ba0ad6693aa/pulsar-client-kafka-compat/pulsar-client-kafka/src/main/java/org/apache/kafka/clients/consumer/PulsarKafkaConsumer.java#L382 ) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d
merlimat commented on issue #3362: java.lang.IllegalArgumentException: Illegal
base64 character 2d
URL: https://github.com/apache/pulsar/issues/3362#issuecomment-453615684
@sautran The code there should be like :
```java
private K getKey(String topic, Message msg) {
if (!msg.hasKey()) {
return null;
}
if (keyDeserializer instanceof StringDeserializer) {
return (K) msg.getKey();
} else {
// Assume base64 encoding
byte[] data = Base64.getDecoder().decode(msg.getKey());
return keyDeserializer.deserialize(topic, data);
}
}
```
Did you set the `StringDeserializer` for the key?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] rdhabalia commented on issue #3348: Remove broker mode to handle persistent/non-persistent topics separately
rdhabalia commented on issue #3348: Remove broker mode to handle persistent/non-persistent topics separately URL: https://github.com/apache/pulsar/pull/3348#issuecomment-453615210 rerun cpp tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sautran opened a new issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d
sautran opened a new issue #3362: java.lang.IllegalArgumentException: Illegal base64 character 2d URL: https://github.com/apache/pulsar/issues/3362 I am using kafka api, the topic is persistent and partitioned topic, in the java consumer, the consumer.poll method throws the following exception java.lang.IllegalArgumentException: Illegal base64 character 2d at java.util.Base64$Decoder.decode0(Base64.java:714) at java.util.Base64$Decoder.decode(Base64.java:526) at java.util.Base64$Decoder.decode(Base64.java:549) at org.apache.kafka.clients.consumer.KafkaConsumer.getKey(KafkaConsumer.java:364) In the producer, the key is "tenant-4", final ProducerRecord data = new ProducerRecord<>( "persistent://public/default/abc", "tenant-4", outStream.toByteArray()); In the pulsar admin api /pulsar-client consume, the message is displayed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated: Added filtering to Twitter Firehose Connector (#3298)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 210d828 Added filtering to Twitter Firehose Connector (#3298)
210d828 is described below
commit 210d828419f8247e6999051c66ee99be544a6221
Author: David Kjerrumgaard <35466513+david-stream...@users.noreply.github.com>
AuthorDate: Fri Jan 11 10:29:36 2019 -0800
Added filtering to Twitter Firehose Connector (#3298)
---
pulsar-io/twitter/pom.xml | 12 +++
.../apache/pulsar/io/twitter/TwitterFireHose.java | 109 +++-
.../pulsar/io/twitter/TwitterFireHoseConfig.java | 88 +++--
.../pulsar/io/twitter/{ => data}/TweetData.java| 24 -
.../pulsar/io/twitter/data/TwitterRecord.java | 66 +
.../pulsar/io/twitter/data/package-info.java | 19
.../twitter/endpoint/SampleStatusesEndpoint.java | 42
.../pulsar/io/twitter/endpoint/package-info.java | 19
.../org/apache/pulsar/io/twitter/package-info.java | 19
.../io/twitter/TwitterFireHoseConfigTests.java | 110 +
.../twitter/src/test/resources/sourceConfig.yaml | 23 +
11 files changed, 443 insertions(+), 88 deletions(-)
diff --git a/pulsar-io/twitter/pom.xml b/pulsar-io/twitter/pom.xml
index cfa3cf6..48ea652 100644
--- a/pulsar-io/twitter/pom.xml
+++ b/pulsar-io/twitter/pom.xml
@@ -53,6 +53,18 @@
hbc-core
${hbc-core.version}
+
+
+ commons-collections
+ commons-collections
+ ${commons.collections.version}
+
+
+
+ org.apache.commons
+ commons-lang3
+ 3.4
+
diff --git
a/pulsar-io/twitter/src/main/java/org/apache/pulsar/io/twitter/TwitterFireHose.java
b/pulsar-io/twitter/src/main/java/org/apache/pulsar/io/twitter/TwitterFireHose.java
index e1d0545..1219e0b 100644
---
a/pulsar-io/twitter/src/main/java/org/apache/pulsar/io/twitter/TwitterFireHose.java
+++
b/pulsar-io/twitter/src/main/java/org/apache/pulsar/io/twitter/TwitterFireHose.java
@@ -24,7 +24,7 @@ import com.fasterxml.jackson.databind.ObjectMapper;
import com.twitter.hbc.ClientBuilder;
import com.twitter.hbc.common.DelimitedStreamReader;
import com.twitter.hbc.core.Constants;
-import com.twitter.hbc.core.endpoint.StatusesSampleEndpoint;
+import com.twitter.hbc.core.endpoint.StatusesFilterEndpoint;
import com.twitter.hbc.core.endpoint.StreamingEndpoint;
import com.twitter.hbc.core.processor.HosebirdMessageProcessor;
import com.twitter.hbc.httpclient.BasicClient;
@@ -33,23 +33,24 @@ import com.twitter.hbc.httpclient.auth.OAuth1;
import java.io.IOException;
import java.io.InputStream;
-import java.io.Serializable;
-import java.text.SimpleDateFormat;
-import java.util.Date;
+import java.util.List;
import java.util.Map;
-import java.util.Optional;
import lombok.extern.slf4j.Slf4j;
-import org.apache.pulsar.functions.api.Record;
+
+import org.apache.commons.collections.CollectionUtils;
import org.apache.pulsar.io.core.PushSource;
import org.apache.pulsar.io.core.SourceContext;
import org.apache.pulsar.io.core.annotations.Connector;
import org.apache.pulsar.io.core.annotations.IOType;
+import org.apache.pulsar.io.twitter.data.TweetData;
+import org.apache.pulsar.io.twitter.data.TwitterRecord;
+import org.apache.pulsar.io.twitter.endpoint.SampleStatusesEndpoint;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * Simple Push based Twitter FireHose Source
+ * Simple Push based Twitter FireHose Source.
*/
@Connector(
name = "twitter",
@@ -62,22 +63,16 @@ public class TwitterFireHose extends PushSource {
private static final Logger LOG =
LoggerFactory.getLogger(TwitterFireHose.class);
-// - Fields set by the constructor
-
// - Runtime fields
private Object waitObject;
-private final ObjectMapper mapper = new
ObjectMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES,
false);
+private final ObjectMapper mapper = new ObjectMapper().configure(
+DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
@Override
public void open(Map config, SourceContext sourceContext)
throws IOException {
TwitterFireHoseConfig hoseConfig = TwitterFireHoseConfig.load(config);
-if (hoseConfig.getConsumerKey() == null
-|| hoseConfig.getConsumerSecret() == null
-|| hoseConfig.getToken() == null
-|| hoseConfig.getTokenSecret() == null) {
-throw new IllegalArgumentException("Required property not set.");
-}
+hoseConfig.validate();
waitObject = new Object();
startThread(hoseConfig);
}
@@ -87,40 +82,13 @@ public class TwitterFireHose extends PushSource {
stopThread();
}
-// -- Custom endpoints
-
-/
[GitHub] merlimat closed pull request #3298: Added filtering to Twitter Firehose Connector
merlimat closed pull request #3298: Added filtering to Twitter Firehose
Connector
URL: https://github.com/apache/pulsar/pull/3298
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/pulsar-io/twitter/pom.xml b/pulsar-io/twitter/pom.xml
index cfa3cf6dc1..48ea6526c2 100644
--- a/pulsar-io/twitter/pom.xml
+++ b/pulsar-io/twitter/pom.xml
@@ -53,6 +53,18 @@
hbc-core
${hbc-core.version}
+
+
+ commons-collections
+ commons-collections
+ ${commons.collections.version}
+
+
+
+ org.apache.commons
+ commons-lang3
+ 3.4
+
diff --git
a/pulsar-io/twitter/src/main/java/org/apache/pulsar/io/twitter/TwitterFireHose.java
b/pulsar-io/twitter/src/main/java/org/apache/pulsar/io/twitter/TwitterFireHose.java
index e1d0545d93..1219e0b19c 100644
---
a/pulsar-io/twitter/src/main/java/org/apache/pulsar/io/twitter/TwitterFireHose.java
+++
b/pulsar-io/twitter/src/main/java/org/apache/pulsar/io/twitter/TwitterFireHose.java
@@ -24,7 +24,7 @@
import com.twitter.hbc.ClientBuilder;
import com.twitter.hbc.common.DelimitedStreamReader;
import com.twitter.hbc.core.Constants;
-import com.twitter.hbc.core.endpoint.StatusesSampleEndpoint;
+import com.twitter.hbc.core.endpoint.StatusesFilterEndpoint;
import com.twitter.hbc.core.endpoint.StreamingEndpoint;
import com.twitter.hbc.core.processor.HosebirdMessageProcessor;
import com.twitter.hbc.httpclient.BasicClient;
@@ -33,23 +33,24 @@
import java.io.IOException;
import java.io.InputStream;
-import java.io.Serializable;
-import java.text.SimpleDateFormat;
-import java.util.Date;
+import java.util.List;
import java.util.Map;
-import java.util.Optional;
import lombok.extern.slf4j.Slf4j;
-import org.apache.pulsar.functions.api.Record;
+
+import org.apache.commons.collections.CollectionUtils;
import org.apache.pulsar.io.core.PushSource;
import org.apache.pulsar.io.core.SourceContext;
import org.apache.pulsar.io.core.annotations.Connector;
import org.apache.pulsar.io.core.annotations.IOType;
+import org.apache.pulsar.io.twitter.data.TweetData;
+import org.apache.pulsar.io.twitter.data.TwitterRecord;
+import org.apache.pulsar.io.twitter.endpoint.SampleStatusesEndpoint;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * Simple Push based Twitter FireHose Source
+ * Simple Push based Twitter FireHose Source.
*/
@Connector(
name = "twitter",
@@ -62,22 +63,16 @@
private static final Logger LOG =
LoggerFactory.getLogger(TwitterFireHose.class);
-// - Fields set by the constructor
-
// - Runtime fields
private Object waitObject;
-private final ObjectMapper mapper = new
ObjectMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES,
false);
+private final ObjectMapper mapper = new ObjectMapper().configure(
+DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
@Override
public void open(Map config, SourceContext sourceContext)
throws IOException {
TwitterFireHoseConfig hoseConfig = TwitterFireHoseConfig.load(config);
-if (hoseConfig.getConsumerKey() == null
-|| hoseConfig.getConsumerSecret() == null
-|| hoseConfig.getToken() == null
-|| hoseConfig.getTokenSecret() == null) {
-throw new IllegalArgumentException("Required property not set.");
-}
+hoseConfig.validate();
waitObject = new Object();
startThread(hoseConfig);
}
@@ -87,40 +82,13 @@ public void close() throws Exception {
stopThread();
}
-// -- Custom endpoints
-
-/**
- * Implementing this interface allows users of this source to set a custom
endpoint.
- */
-public interface EndpointInitializer {
-StreamingEndpoint createEndpoint();
-}
-
-/**
- * Required for Twitter Client
- */
-private static class SampleStatusesEndpoint implements
EndpointInitializer, Serializable {
-@Override
-public StreamingEndpoint createEndpoint() {
-// this default endpoint initializer returns the sample endpoint:
Returning a sample from the firehose (all tweets)
-StatusesSampleEndpoint endpoint = new StatusesSampleEndpoint();
-endpoint.stallWarnings(false);
-endpoint.delimited(false);
-return endpoint;
-}
-}
-
private void startThread(TwitterFireHoseConfig config) {
-Authentication auth = new OAuth1(config.getConsumerKey(),
-config.getConsumerSecret(),
-config.getToken(),
-config.getTokenSecret());
BasicClient client = new ClientBuilder()
.name(config.getClientName())
[pulsar] branch master updated: Fix title and content mismatch issue in cookbooks-partitioned.md (#3349) (#3353)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new aa20363 Fix title and content mismatch issue in cookbooks-partitioned.md (#3349) (#3353) aa20363 is described below commit aa2036309c4804c08976675ecdf7f2abcf0ddd0e Author: Vincent Xie AuthorDate: Sat Jan 12 02:28:22 2019 +0800 Fix title and content mismatch issue in cookbooks-partitioned.md (#3349) (#3353) * Fix title and content mismatch issue in cookbooks-partitioned.md (#3349) Signed-off-by: Vincent Xie * Update site2/docs/cookbooks-partitioned.md Co-Authored-By: VinceShieh --- site2/docs/cookbooks-partitioned.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site2/docs/cookbooks-partitioned.md b/site2/docs/cookbooks-partitioned.md index 182299f..76b6ed6 100644 --- a/site2/docs/cookbooks-partitioned.md +++ b/site2/docs/cookbooks-partitioned.md @@ -1,6 +1,6 @@ --- id: cookbooks-partitioned -title: Non-persistent messaging +title: Partitioned topics sidebar_label: Partitioned Topics ---
[GitHub] merlimat closed pull request #3353: Fix title and content mismatch issue in cookbooks-partitioned.md (#3349)
merlimat closed pull request #3353: Fix title and content mismatch issue in cookbooks-partitioned.md (#3349) URL: https://github.com/apache/pulsar/pull/3353 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/site2/docs/cookbooks-partitioned.md b/site2/docs/cookbooks-partitioned.md index 182299f0e0..76b6ed657b 100644 --- a/site2/docs/cookbooks-partitioned.md +++ b/site2/docs/cookbooks-partitioned.md @@ -1,6 +1,6 @@ --- id: cookbooks-partitioned -title: Non-persistent messaging +title: Partitioned topics sidebar_label: Partitioned Topics --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat closed pull request #3301: Minor typo (cd instead of cp)
merlimat closed pull request #3301: Minor typo (cd instead of cp)
URL: https://github.com/apache/pulsar/pull/3301
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/site2/docs/getting-started-standalone.md
b/site2/docs/getting-started-standalone.md
index 9557939bec..91214add95 100644
--- a/site2/docs/getting-started-standalone.md
+++ b/site2/docs/getting-started-standalone.md
@@ -86,7 +86,7 @@ $ tar xvfz
/path/to/apache-pulsar-io-connectors-{{pulsar:version}}-bin.tar.gz
// you will find a directory named
`apache-pulsar-io-connectors-{{pulsar:version}}` in the pulsar directory
// then copy the connectors
-$ cd apache-pulsar-io-connectors-{{pulsar:version}}/connectors connectors
+$ cp -r apache-pulsar-io-connectors-{{pulsar:version}}/connectors connectors
$ ls connectors
pulsar-io-aerospike-{{pulsar:version}}.nar
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[pulsar] branch master updated (aa20363 -> b14aca8)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git. from aa20363 Fix title and content mismatch issue in cookbooks-partitioned.md (#3349) (#3353) add b14aca8 Minor typo (cd instead of cp) (#3301) No new revisions were added by this update. Summary of changes: site2/docs/getting-started-standalone.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[pulsar] branch master updated: Integration test and documentation for Proxy w/o ZK (#3352)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/master by this push: new aa3fd10 Integration test and documentation for Proxy w/o ZK (#3352) aa3fd10 is described below commit aa3fd10c87f6b2cc0a3d57dd67b4e205bb49a0d0 Author: Ivan Kelly AuthorDate: Fri Jan 11 19:27:54 2019 +0100 Integration test and documentation for Proxy w/o ZK (#3352) Adds an integration test which creates a proxy which connects to the brokers using broker URLs rather than service discovery. Adds documentation for configuring the proxy in this way. Issue: #2405 --- site2/docs/administration-proxy.md | 49 +++--- site2/docs/security-authorization.md | 4 +- .../pulsar/tests/integration/proxy/TestProxy.java | 42 +++ .../integration/topologies/PulsarClusterSpec.java | 2 + 4 files changed, 83 insertions(+), 14 deletions(-) diff --git a/site2/docs/administration-proxy.md b/site2/docs/administration-proxy.md index 3321e80..75efdcb 100644 --- a/site2/docs/administration-proxy.md +++ b/site2/docs/administration-proxy.md @@ -4,19 +4,56 @@ title: The Pulsar proxy sidebar_label: Pulsar proxy --- -The [Pulsar proxy](concepts-architecture-overview.md#pulsar-proxy) is an optional gateway that you can run over the brokers in a Pulsar cluster. We recommend running a Pulsar proxy in cases when direction connections between clients and Pulsar brokers are either infeasible, undesirable, or both, for example when running Pulsar in a cloud environment or on [Kubernetes](https://kubernetes.io) or an analogous platform. +The [Pulsar proxy](concepts-architecture-overview.md#pulsar-proxy) is an optional gateway that you can run in front of the brokers in a Pulsar cluster. We recommend running a Pulsar proxy in cases when direction connections between clients and Pulsar brokers are either infeasible, undesirable, or both, for example when running Pulsar in a cloud environment or on [Kubernetes](https://kubernetes.io) or an analogous platform. -## Running the proxy +## Configuring the proxy -In order to run the Pulsar proxy, you need to have both a local [ZooKeeper](https://zookeeper.apache.org) and configuration store quorum set up for use by your Pulsar cluster. For instructions, see [this document](deploy-bare-metal.md). Once you have ZooKeeper set up and have connection strings for both ZooKeeper quorums, you can use the [`proxy`](reference-cli-tools.md#pulsar-proxy) command of the [`pulsar`](reference-cli-tools.md#pulsar) CLI tool to start up the proxy (preferably on it [...] +The proxy must have some way to find the addresses of the brokers of the cluster. You can do this by either configuring the proxy to connect directly to service discovery or by specifying a broker URL in the configuration. + +### Option 1: Using service discovery + +Pulsar uses [ZooKeeper](https://zookeeper.apache.org) for service discovery. To connect the proxy to ZooKeeper, specify the following in `conf/proxy.conf`. +```properties +zookeeperServers=zk-0,zk-1,zk-2 +configurationStoreServers=zk-0:2184,zk-remote:2184 +``` + +> If using service discovery, the network ACL must allow the proxy to talk to the ZooKeeper nodes on the zookeeper client port, which is usually 2181, and on the configuration store client port, which is 2184 by default. Opening the network ACLs means that if someone compromises a proxy, they have full access to ZooKeeper. For this reason, it is more secure to use broker URLs to configure the proxy. + +### Option 2: Using broker URLs + +The more secure method of configuring the proxy is to specify a URL to connect to the brokers. + +> [Authorization](security-authorization#enabling-authorization-and-assigning-superusers) at the proxy requires access to ZooKeeper, so if you are using this broker URLs to connect to the brokers, Proxy level authorization should be disabled. Brokers will still authorize requests after the proxy forwards them. + +You can configure the broker URLs in `conf/proxy.conf` as follows. + +```properties +brokerServiceURL=pulsar://brokers.example.com:6650 +brokerWebServiceURL=http://brokers.example.com:8080 +functionWorkerWebServiceURL=http://function-workers.example.com:8080 +``` + +Or if using TLS: +```properties +brokerServiceURLTLS=pulsar+ssl://brokers.example.com:6651 +brokerWebServiceURLTLS=https://brokers.example.com:8443 +functionWorkerWebServiceURL=https://function-workers.example.com:8443 +``` + +The hostname in the URLs provided should be a DNS entry which points to multiple brokers or a Virtual IP which is backed by multiple broker IP addresses so that the proxy does not lose connectivity to the pulsar cluster if a single broker becomes unavailable. + +The ports to connect to the brokers (6650 & 8080, or in the
[GitHub] merlimat closed pull request #3352: Integration test and documentation for Proxy w/o ZK
merlimat closed pull request #3352: Integration test and documentation for Proxy w/o ZK URL: https://github.com/apache/pulsar/pull/3352 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/site2/docs/administration-proxy.md b/site2/docs/administration-proxy.md index 3321e80cb3..75efdcb45b 100644 --- a/site2/docs/administration-proxy.md +++ b/site2/docs/administration-proxy.md @@ -4,19 +4,56 @@ title: The Pulsar proxy sidebar_label: Pulsar proxy --- -The [Pulsar proxy](concepts-architecture-overview.md#pulsar-proxy) is an optional gateway that you can run over the brokers in a Pulsar cluster. We recommend running a Pulsar proxy in cases when direction connections between clients and Pulsar brokers are either infeasible, undesirable, or both, for example when running Pulsar in a cloud environment or on [Kubernetes](https://kubernetes.io) or an analogous platform. +The [Pulsar proxy](concepts-architecture-overview.md#pulsar-proxy) is an optional gateway that you can run in front of the brokers in a Pulsar cluster. We recommend running a Pulsar proxy in cases when direction connections between clients and Pulsar brokers are either infeasible, undesirable, or both, for example when running Pulsar in a cloud environment or on [Kubernetes](https://kubernetes.io) or an analogous platform. -## Running the proxy +## Configuring the proxy -In order to run the Pulsar proxy, you need to have both a local [ZooKeeper](https://zookeeper.apache.org) and configuration store quorum set up for use by your Pulsar cluster. For instructions, see [this document](deploy-bare-metal.md). Once you have ZooKeeper set up and have connection strings for both ZooKeeper quorums, you can use the [`proxy`](reference-cli-tools.md#pulsar-proxy) command of the [`pulsar`](reference-cli-tools.md#pulsar) CLI tool to start up the proxy (preferably on its own machine or in its own VM): +The proxy must have some way to find the addresses of the brokers of the cluster. You can do this by either configuring the proxy to connect directly to service discovery or by specifying a broker URL in the configuration. + +### Option 1: Using service discovery + +Pulsar uses [ZooKeeper](https://zookeeper.apache.org) for service discovery. To connect the proxy to ZooKeeper, specify the following in `conf/proxy.conf`. +```properties +zookeeperServers=zk-0,zk-1,zk-2 +configurationStoreServers=zk-0:2184,zk-remote:2184 +``` + +> If using service discovery, the network ACL must allow the proxy to talk to the ZooKeeper nodes on the zookeeper client port, which is usually 2181, and on the configuration store client port, which is 2184 by default. Opening the network ACLs means that if someone compromises a proxy, they have full access to ZooKeeper. For this reason, it is more secure to use broker URLs to configure the proxy. + +### Option 2: Using broker URLs + +The more secure method of configuring the proxy is to specify a URL to connect to the brokers. + +> [Authorization](security-authorization#enabling-authorization-and-assigning-superusers) at the proxy requires access to ZooKeeper, so if you are using this broker URLs to connect to the brokers, Proxy level authorization should be disabled. Brokers will still authorize requests after the proxy forwards them. + +You can configure the broker URLs in `conf/proxy.conf` as follows. + +```properties +brokerServiceURL=pulsar://brokers.example.com:6650 +brokerWebServiceURL=http://brokers.example.com:8080 +functionWorkerWebServiceURL=http://function-workers.example.com:8080 +``` + +Or if using TLS: +```properties +brokerServiceURLTLS=pulsar+ssl://brokers.example.com:6651 +brokerWebServiceURLTLS=https://brokers.example.com:8443 +functionWorkerWebServiceURL=https://function-workers.example.com:8443 +``` + +The hostname in the URLs provided should be a DNS entry which points to multiple brokers or a Virtual IP which is backed by multiple broker IP addresses so that the proxy does not lose connectivity to the pulsar cluster if a single broker becomes unavailable. + +The ports to connect to the brokers (6650 & 8080, or in the case of TLS, 6651 & 8443) should be open in the network ACLs. + +Note that if you are not using functions, then `functionWorkerWebServiceURL` does not need to be configured. + +## Starting the proxy To start the proxy: ```bash $ cd /path/to/pulsar/directory -$ bin/pulsar proxy \ - --zookeeper-servers zk-0,zk-1,zk-2 \ - --global-zookeeper-servers zk-0,zk-1,zk-2 +$ bin/pulsar proxy ``` > You can run as many instances of the Pulsar proxy in a cluster as you would > like. diff --git a/site2/docs/security-authorization.md b/site2/docs/security-authorization.md index 2cd168a6c1..fe26cd7da7 100644 --- a/site2/docs/securi
[GitHub] merlimat commented on issue #3357: fix dep to make build compatible with jdk11
merlimat commented on issue #3357: fix dep to make build compatible with jdk11 URL: https://github.com/apache/pulsar/pull/3357#issuecomment-453611773 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #3358: Fix bug consumer stop consume(issue-3131)
merlimat commented on issue #3358: Fix bug consumer stop consume(issue-3131) URL: https://github.com/apache/pulsar/pull/3358#issuecomment-453611577 @codelipenghui The change LGTM. Is it possible to reproduce the issue within a unit test? That would be great to have a better understanding of this condition. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sautran commented on issue #3361: Kafka configure properties are not included subscription type and subscription mode
sautran commented on issue #3361: Kafka configure properties are not included subscription type and subscription mode URL: https://github.com/apache/pulsar/issues/3361#issuecomment-453611424 @merlimat Great! Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated: Pass schema info to C++ client (#3354)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new ef94cb7 Pass schema info to C++ client (#3354)
ef94cb7 is described below
commit ef94cb70df08d7e9f16c6415116e729bb6d756ab
Author: Matteo Merli
AuthorDate: Fri Jan 11 10:25:24 2019 -0800
Pass schema info to C++ client (#3354)
* Pass schema info to C++ client
* Comments
---
pulsar-client-cpp/include/pulsar/Client.h | 1 +
.../include/pulsar/ConsumerConfiguration.h | 16 +++
pulsar-client-cpp/include/pulsar/Producer.h| 10 ++
.../include/pulsar/ProducerConfiguration.h | 21 +++
.../include/pulsar/ReaderConfiguration.h | 16 +++
pulsar-client-cpp/include/pulsar/Schema.h | 160 +
pulsar-client-cpp/lib/ClientConnection.cc | 11 +-
pulsar-client-cpp/lib/ClientConnection.h | 7 +-
pulsar-client-cpp/lib/Commands.cc | 61 +++-
pulsar-client-cpp/lib/Commands.h | 7 +-
pulsar-client-cpp/lib/ConsumerConfiguration.cc | 7 +
pulsar-client-cpp/lib/ConsumerConfigurationImpl.h | 4 +-
pulsar-client-cpp/lib/ConsumerImpl.cc | 6 +-
pulsar-client-cpp/lib/PartitionedProducerImpl.cc | 6 +
pulsar-client-cpp/lib/PartitionedProducerImpl.h| 2 +
pulsar-client-cpp/lib/Producer.cc | 2 +
pulsar-client-cpp/lib/ProducerConfiguration.cc | 7 +
pulsar-client-cpp/lib/ProducerConfigurationImpl.h | 4 +-
pulsar-client-cpp/lib/ProducerImpl.cc | 13 +-
pulsar-client-cpp/lib/ProducerImpl.h | 3 +
pulsar-client-cpp/lib/ProducerImplBase.h | 1 +
pulsar-client-cpp/lib/ReaderConfiguration.cc | 7 +
pulsar-client-cpp/lib/ReaderConfigurationImpl.h| 4 +-
pulsar-client-cpp/lib/ReaderImpl.cc| 1 +
pulsar-client-cpp/lib/Schema.cc| 103 +
pulsar-client-cpp/tests/SchemaTest.cc | 69 +
26 files changed, 529 insertions(+), 20 deletions(-)
diff --git a/pulsar-client-cpp/include/pulsar/Client.h
b/pulsar-client-cpp/include/pulsar/Client.h
index d77eb6d..e04b1b8 100644
--- a/pulsar-client-cpp/include/pulsar/Client.h
+++ b/pulsar-client-cpp/include/pulsar/Client.h
@@ -26,6 +26,7 @@
#include
#include
#include
+#include
#include
#pragma GCC visibility push(default)
diff --git a/pulsar-client-cpp/include/pulsar/ConsumerConfiguration.h
b/pulsar-client-cpp/include/pulsar/ConsumerConfiguration.h
index 0687166..60ffef1 100644
--- a/pulsar-client-cpp/include/pulsar/ConsumerConfiguration.h
+++ b/pulsar-client-cpp/include/pulsar/ConsumerConfiguration.h
@@ -24,6 +24,7 @@
#include
#include
#include
+#include
#include
#include
@@ -52,6 +53,21 @@ class ConsumerConfiguration {
ConsumerConfiguration& operator=(const ConsumerConfiguration&);
/**
+ * Declare the schema of the data that this consumer will be accepting.
+ *
+ * The schema will be checked against the schema of the topic, and the
+ * consumer creation will fail if it's not compatible.
+ *
+ * @param schemaInfo the schema definition object
+ */
+ConsumerConfiguration& setSchema(const SchemaInfo& schemaInfo);
+
+/**
+ * @return the schema information declared for this consumer
+ */
+const SchemaInfo& getSchema() const;
+
+/**
* Specify the consumer type. The consumer type enables
* specifying the type of subscription. In Exclusive subscription,
* only a single consumer is allowed to attach to the subscription. Other
consumers
diff --git a/pulsar-client-cpp/include/pulsar/Producer.h
b/pulsar-client-cpp/include/pulsar/Producer.h
index 407d937..1941e15 100644
--- a/pulsar-client-cpp/include/pulsar/Producer.h
+++ b/pulsar-client-cpp/include/pulsar/Producer.h
@@ -110,6 +110,16 @@ class Producer {
int64_t getLastSequenceId() const;
/**
+ * Return an identifier for the schema version that this producer was
created with.
+ *
+ * When the producer is created, if a schema info was passed, the broker
will
+ * determine the version of the passed schema. This identifier should be
treated
+ * as an opaque identifier. In particular, even though this is represented
as a string, the
+ * version might not be ascii printable.
+ */
+const std::string& getSchemaVersion() const;
+
+/**
* Close the producer and release resources allocated.
*
* No more writes will be accepted from this producer. Waits until
diff --git a/pulsar-client-cpp/include/pulsar/ProducerConfiguration.h
b/pulsar-client-cpp/include/pulsar/ProducerConfiguration.h
index 565a6ab..6e3d0b4 100644
--- a/pulsar-client-cpp/include/pulsar/ProducerConfiguration.h
+++ b/pulsar-client-cpp/include/pulsar/
[GitHub] merlimat closed pull request #3354: Pass schema info to C++ client
merlimat closed pull request #3354: Pass schema info to C++ client
URL: https://github.com/apache/pulsar/pull/3354
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/pulsar-client-cpp/include/pulsar/Client.h
b/pulsar-client-cpp/include/pulsar/Client.h
index d77eb6d9f5..e04b1b88b7 100644
--- a/pulsar-client-cpp/include/pulsar/Client.h
+++ b/pulsar-client-cpp/include/pulsar/Client.h
@@ -26,6 +26,7 @@
#include
#include
#include
+#include
#include
#pragma GCC visibility push(default)
diff --git a/pulsar-client-cpp/include/pulsar/ConsumerConfiguration.h
b/pulsar-client-cpp/include/pulsar/ConsumerConfiguration.h
index 0687166d74..60ffef1f85 100644
--- a/pulsar-client-cpp/include/pulsar/ConsumerConfiguration.h
+++ b/pulsar-client-cpp/include/pulsar/ConsumerConfiguration.h
@@ -24,6 +24,7 @@
#include
#include
#include
+#include
#include
#include
@@ -51,6 +52,21 @@ class ConsumerConfiguration {
ConsumerConfiguration(const ConsumerConfiguration&);
ConsumerConfiguration& operator=(const ConsumerConfiguration&);
+/**
+ * Declare the schema of the data that this consumer will be accepting.
+ *
+ * The schema will be checked against the schema of the topic, and the
+ * consumer creation will fail if it's not compatible.
+ *
+ * @param schemaInfo the schema definition object
+ */
+ConsumerConfiguration& setSchema(const SchemaInfo& schemaInfo);
+
+/**
+ * @return the schema information declared for this consumer
+ */
+const SchemaInfo& getSchema() const;
+
/**
* Specify the consumer type. The consumer type enables
* specifying the type of subscription. In Exclusive subscription,
diff --git a/pulsar-client-cpp/include/pulsar/Producer.h
b/pulsar-client-cpp/include/pulsar/Producer.h
index 407d937043..1941e15923 100644
--- a/pulsar-client-cpp/include/pulsar/Producer.h
+++ b/pulsar-client-cpp/include/pulsar/Producer.h
@@ -109,6 +109,16 @@ class Producer {
*/
int64_t getLastSequenceId() const;
+/**
+ * Return an identifier for the schema version that this producer was
created with.
+ *
+ * When the producer is created, if a schema info was passed, the broker
will
+ * determine the version of the passed schema. This identifier should be
treated
+ * as an opaque identifier. In particular, even though this is represented
as a string, the
+ * version might not be ascii printable.
+ */
+const std::string& getSchemaVersion() const;
+
/**
* Close the producer and release resources allocated.
*
diff --git a/pulsar-client-cpp/include/pulsar/ProducerConfiguration.h
b/pulsar-client-cpp/include/pulsar/ProducerConfiguration.h
index 565a6ab0ea..6e3d0b4c01 100644
--- a/pulsar-client-cpp/include/pulsar/ProducerConfiguration.h
+++ b/pulsar-client-cpp/include/pulsar/ProducerConfiguration.h
@@ -25,6 +25,7 @@
#include
#include
#include
+#include
#include
@@ -55,6 +56,7 @@ class ProducerConfiguration {
BoostHash,
JavaStringHash
};
+
ProducerConfiguration();
~ProducerConfiguration();
ProducerConfiguration(const ProducerConfiguration&);
@@ -63,6 +65,25 @@ class ProducerConfiguration {
ProducerConfiguration& setProducerName(const std::string& producerName);
const std::string& getProducerName() const;
+/**
+ * Declare the schema of the data that will be published by this producer.
+ *
+ * The schema will be checked against the schema of the topic, and it
+ * will fail if it's not compatible, though the client library will
+ * not perform any validation that the actual message payload are
+ * conforming to the specified schema.
+ *
+ * For all purposes, this
+ * @param schemaInfo
+ * @return
+ */
+ProducerConfiguration& setSchema(const SchemaInfo& schemaInfo);
+
+/**
+ * @return the schema information declared for this producer
+ */
+const SchemaInfo& getSchema() const;
+
ProducerConfiguration& setSendTimeout(int sendTimeoutMs);
int getSendTimeout() const;
diff --git a/pulsar-client-cpp/include/pulsar/ReaderConfiguration.h
b/pulsar-client-cpp/include/pulsar/ReaderConfiguration.h
index 8d365ab813..69776e32ef 100644
--- a/pulsar-client-cpp/include/pulsar/ReaderConfiguration.h
+++ b/pulsar-client-cpp/include/pulsar/ReaderConfiguration.h
@@ -23,6 +23,7 @@
#include
#include
#include
+#include
#pragma GCC visibility push(default)
namespace pulsar {
@@ -48,6 +49,21 @@ class ReaderConfiguration {
ReaderConfiguration(const ReaderConfiguration&);
ReaderConfiguration& operator=(const ReaderConfiguration&);
+/**
+ * Declare the schema of the data that this reader will be accepting.
+
[GitHub] merlimat commented on issue #3361: Kafka configure properties are not included subscription type and subscription mode
merlimat commented on issue #3361: Kafka configure properties are not included subscription type and subscription mode URL: https://github.com/apache/pulsar/issues/3361#issuecomment-453610968 @sautran That's correct. A "failover" subscription over a partitioned topic will work in a similar way as of a Kafka consumer group. That's the reason we hardcode the the subscription type to "failover" in the Kafka client wrapper. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sautran commented on issue #3361: Kafka configure properties are not included subscription type and subscription mode
sautran commented on issue #3361: Kafka configure properties are not included subscription type and subscription mode URL: https://github.com/apache/pulsar/issues/3361#issuecomment-453590038 For the subscriptionType, right now our kafka application have a few consumers using the same group.id on a partitioned topic. Each consumer reads some partitions. My question is in Pulsar, if one consumer die, are other consumers taking those partitions over? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] merlimat commented on issue #3361: Kafka configure properties are not included subscription type and subscription mode
merlimat commented on issue #3361: Kafka configure properties are not included subscription type and subscription mode URL: https://github.com/apache/pulsar/issues/3361#issuecomment-453587691 I agree that `subscriptionTopicsMode` should be configurable. Though I believe `subscriptionType` can only be `Failover`, since Kafka API has no option to do individual acknowledge of messages. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sautran opened a new issue #3361: Kafka configure properties are not included subscription type and subscription mode
sautran opened a new issue #3361: Kafka configure properties are not included subscription type and subscription mode URL: https://github.com/apache/pulsar/issues/3361 I am trying to use the latest pulsar release to migrate current kafka client to pulsar and try to use the kafka pulsar wrapper. I am using non-persistent topic, in the kafka pulsar client config, Following is the default "subscriptionTopicsMode" : "PERSISTENT", "subscriptionType" : "Failover", Is there anyway to set subscriptionTopicsMode and subscriptionType through the kafka configuration properties map? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] thatcort commented on issue #1070: Provide a nodejs client
thatcort commented on issue #1070: Provide a nodejs client URL: https://github.com/apache/pulsar/issues/1070#issuecomment-453548857 Just a suggestion to use TypeScript, rather than JavaScript. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source submitted with invalid configuration stucks without consuming from Kafka topics
ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source
submitted with invalid configuration stucks without consuming from Kafka topics
URL: https://github.com/apache/pulsar/issues/3294#issuecomment-453418838
@sijie
Use 'source create' command we validate the parameter key is included, but
it does not verify that the value of the key is correct, then create source's
calss instance, not use
[Source.open()](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java#L55)
method, so create sucessflully.
If value is error, should we return the create source failure?
kafka.yaml info:
```file
configs:
bootstrapServers: "xxx"
groupId: "pulsarUTset"
topic: "test_kafka_topic"
fetchMinBytes: 1
autoCommitEnabled: "true"
autoCommitIntervalMs: 1000
sessionTimeoutMs: 3000
```
or
```file
configs:
```
command like :
```shell
./bin/pulsar-admin source create --source-config-file
~/code/pulsar/pulsar_ms/file/kafka.yaml -t kafka --destination-topic-name
dest_topic --name kafka_fn
"Created successfully"
```
when I getstatus return info:
```shell
$ ./bin/pulsar-admin source getstatus --name kafka_fn
{
"numInstances" : 1,
"numRunning" : 1,
"instances" : [ {
"instanceId" : 0,
"status" : {
"running" : true,
"error" : "",
"numRestarts" : 0,
"numReceivedFromSource" : 0,
"numSystemExceptions" : 0,
"latestSystemExceptions" : [ ],
"numSourceExceptions" : 0,
"latestSourceExceptions" : [ ],
"numWritten" : 0,
"lastReceivedTime" : 0,
"workerId" : "c-standalone-fw-localhost-8080"
}
} ]
}
```
Maybe this problem with the content of the pulsar-io module.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source submitted with invalid configuration stucks without consuming from Kafka topics
ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source
submitted with invalid configuration stucks without consuming from Kafka topics
URL: https://github.com/apache/pulsar/issues/3294#issuecomment-453418838
@sijie
Use 'source create' command we validate the parameter key is included, but
it does not verify that the value of the key is correct, then create source's
calss instance, not use
[Source.open()](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java#L55)
method, so create sucessflully.
If value is error, should we return the create source failure?
kafka.yaml info:
```file
configs:
bootstrapServers: "xxx"
groupId: "pulsarUTset"
topic: "test_kafka_topic"
fetchMinBytes: 1
autoCommitEnabled: "true"
autoCommitIntervalMs: 1000
sessionTimeoutMs: 3000
```
or
```file
configs:
```
command like :
```shell
./bin/pulsar-admin source create --source-config-file
~/code/pulsar/pulsar_ms/file/kafka.yaml -t kafka --destination-topic-name
dest_topic --name kafka_fn
"Created successfully"
```
when I getstatus return info:
```shell
$ ./bin/pulsar-admin source getstatus --name kafka_fn
{
"numInstances" : 1,
"numRunning" : 1,
"instances" : [ {
"instanceId" : 0,
"status" : {
"running" : true,
"error" : "",
"numRestarts" : 0,
"numReceivedFromSource" : 0,
"numSystemExceptions" : 0,
"latestSystemExceptions" : [ ],
"numSourceExceptions" : 0,
"latestSourceExceptions" : [ ],
"numWritten" : 0,
"lastReceivedTime" : 0,
"workerId" : "c-standalone-fw-localhost-8080"
}
} ]
}
```
If so,this problem with the content of the pulsar-io module.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] tuteng opened a new pull request #3360: Support TLS authentication and authorization in standalone mode
tuteng opened a new pull request #3360: Support TLS authentication and authorization in standalone mode URL: https://github.com/apache/pulsar/pull/3360 ### Motivation I found that in standalone mode, opening TLS authentication would be overwritten.Opening TLS authentication and authorization in standalone mode is for automated testing. ### Modifications Support TLS authentication and authorization in standalone mode ### Result Test pass. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source submitted with invalid configuration stucks without consuming from Kafka topics
ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source
submitted with invalid configuration stucks without consuming from Kafka topics
URL: https://github.com/apache/pulsar/issues/3294#issuecomment-453418838
@sijie
Use 'source create' command we validate the parameter key is included, but
it does not verify that the value of the key is correct, then new source's
calss instance, not use
[Source.open()](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java#L55)
method, so create sucessflully.
If value is error, should we return the create source failure?
kafka.yaml info:
```file
configs:
bootstrapServers: "xxx"
groupId: "pulsarUTset"
topic: "test_kafka_topic"
fetchMinBytes: 1
autoCommitEnabled: "true"
autoCommitIntervalMs: 1000
sessionTimeoutMs: 3000
```
or
```file
configs:
```
command like :
```shell
./bin/pulsar-admin source create --source-config-file
~/code/pulsar/pulsar_ms/file/kafka.yaml -t kafka --destination-topic-name
dest_topic --name kafka_fn
"Created successfully"
```
when I getstatus return info:
```shell
$ ./bin/pulsar-admin source getstatus --name kafka_fn
{
"numInstances" : 1,
"numRunning" : 1,
"instances" : [ {
"instanceId" : 0,
"status" : {
"running" : true,
"error" : "",
"numRestarts" : 0,
"numReceivedFromSource" : 0,
"numSystemExceptions" : 0,
"latestSystemExceptions" : [ ],
"numSourceExceptions" : 0,
"latestSourceExceptions" : [ ],
"numWritten" : 0,
"lastReceivedTime" : 0,
"workerId" : "c-standalone-fw-localhost-8080"
}
} ]
}
```
If so,this problem with the content of the pulsar-io module.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source submitted with invalid configuration stucks without consuming from Kafka topics
ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source
submitted with invalid configuration stucks without consuming from Kafka topics
URL: https://github.com/apache/pulsar/issues/3294#issuecomment-453418838
@merlimat @sijie
Use 'source create' command we validate the parameter key is included, but
it does not verify that the value of the key is correct, then new source's
calss instance, not use
[Source.open()](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java#L55)
method, so create sucessflully.
If value is error, should we return the create source failure?
kafka.yaml info:
```file
configs:
bootstrapServers: "xxx"
groupId: "pulsarUTset"
topic: "test_kafka_topic"
fetchMinBytes: 1
autoCommitEnabled: "true"
autoCommitIntervalMs: 1000
sessionTimeoutMs: 3000
```
or
```file
configs:
```
command like :
```shell
./bin/pulsar-admin source create --source-config-file
~/code/pulsar/pulsar_ms/file/kafka.yaml -t kafka --destination-topic-name
dest_topic --name kafka_fn
"Created successfully"
```
when I getstatus return info:
```shell
$ ./bin/pulsar-admin source getstatus --name kafka_fn
{
"numInstances" : 1,
"numRunning" : 1,
"instances" : [ {
"instanceId" : 0,
"status" : {
"running" : true,
"error" : "",
"numRestarts" : 0,
"numReceivedFromSource" : 0,
"numSystemExceptions" : 0,
"latestSystemExceptions" : [ ],
"numSourceExceptions" : 0,
"latestSourceExceptions" : [ ],
"numWritten" : 0,
"lastReceivedTime" : 0,
"workerId" : "c-standalone-fw-localhost-8080"
}
} ]
}
```
If so,this problem with the content of the pulsar-io module.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source submitted with invalid configuration stucks without consuming from Kafka topics
ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source
submitted with invalid configuration stucks without consuming from Kafka topics
URL: https://github.com/apache/pulsar/issues/3294#issuecomment-453418838
@merlimat @sijie
Use 'source create' command we validate the parameter key is included, but
it does not verify that the value of the key is correct, just new source's
calss instance, not use
[Source.open()](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java#L55)
method, so create sucessflully.
If value is error, should we return the create source failure?
kafka.yaml info:
```file
configs:
bootstrapServers: "xxx"
groupId: "pulsarUTset"
topic: "test_kafka_topic"
fetchMinBytes: 1
autoCommitEnabled: "true"
autoCommitIntervalMs: 1000
sessionTimeoutMs: 3000
```
or
```file
configs:
```
command like :
```shell
./bin/pulsar-admin source create --source-config-file
~/code/pulsar/pulsar_ms/file/kafka.yaml -t kafka --destination-topic-name
dest_topic --name kafka_fn
"Created successfully"
```
when I getstatus return info:
```shell
$ ./bin/pulsar-admin source getstatus --name kafka_fn
{
"numInstances" : 1,
"numRunning" : 1,
"instances" : [ {
"instanceId" : 0,
"status" : {
"running" : true,
"error" : "",
"numRestarts" : 0,
"numReceivedFromSource" : 0,
"numSystemExceptions" : 0,
"latestSystemExceptions" : [ ],
"numSourceExceptions" : 0,
"latestSourceExceptions" : [ ],
"numWritten" : 0,
"lastReceivedTime" : 0,
"workerId" : "c-standalone-fw-localhost-8080"
}
} ]
}
```
If so,this problem with the content of the pulsar-io module.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source submitted with invalid configuration stucks without consuming from Kafka topics
ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source
submitted with invalid configuration stucks without consuming from Kafka topics
URL: https://github.com/apache/pulsar/issues/3294#issuecomment-453418838
@merlimat @sijie
Use 'source create' command we validate the parameter key is included, but
it does not verify that the value of the key is correct, so create
sucessflully, just new source's calss instance, not use
[Source.open()](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java#L55)
method.
If value is error, should we return the create source failure?
kafka.yaml info:
```file
configs:
bootstrapServers: "xxx"
groupId: "pulsarUTset"
topic: "test_kafka_topic"
fetchMinBytes: 1
autoCommitEnabled: "true"
autoCommitIntervalMs: 1000
sessionTimeoutMs: 3000
```
or
```file
configs:
```
command like :
```shell
./bin/pulsar-admin source create --source-config-file
~/code/pulsar/pulsar_ms/file/kafka.yaml -t kafka --destination-topic-name
dest_topic --name kafka_fn
"Created successfully"
```
when I getstatus return info:
```shell
$ ./bin/pulsar-admin source getstatus --name kafka_fn
{
"numInstances" : 1,
"numRunning" : 1,
"instances" : [ {
"instanceId" : 0,
"status" : {
"running" : true,
"error" : "",
"numRestarts" : 0,
"numReceivedFromSource" : 0,
"numSystemExceptions" : 0,
"latestSystemExceptions" : [ ],
"numSourceExceptions" : 0,
"latestSourceExceptions" : [ ],
"numWritten" : 0,
"lastReceivedTime" : 0,
"workerId" : "c-standalone-fw-localhost-8080"
}
} ]
}
```
If so,this problem with the content of the pulsar-io module.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source submitted with invalid configuration stucks without consuming from Kafka topics
ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source
submitted with invalid configuration stucks without consuming from Kafka topics
URL: https://github.com/apache/pulsar/issues/3294#issuecomment-453418838
Use 'source create' command we validate the parameter key is included, but
it does not verify that the value of the key is correct, so create
sucessflully, just new source's calss instance, not use
[Source.open()](https://github.com/apache/pulsar/blob/master/pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaAbstractSource.java#L55)
method.
If value is error, should we return the create source failure?
kafka.yaml info:
```file
configs:
bootstrapServers: "xxx"
groupId: "pulsarUTset"
topic: "test_kafka_topic"
fetchMinBytes: 1
autoCommitEnabled: "true"
autoCommitIntervalMs: 1000
sessionTimeoutMs: 3000
```
or
```file
configs:
```
command like :
```shell
./bin/pulsar-admin source create --source-config-file
~/code/pulsar/pulsar_ms/file/kafka.yaml -t kafka --destination-topic-name
dest_topic --name kafka_fn
"Created successfully"
```
when I getstatus return info:
```shell
$ ./bin/pulsar-admin source getstatus --name kafka_fn
{
"numInstances" : 1,
"numRunning" : 1,
"instances" : [ {
"instanceId" : 0,
"status" : {
"running" : true,
"error" : "",
"numRestarts" : 0,
"numReceivedFromSource" : 0,
"numSystemExceptions" : 0,
"latestSystemExceptions" : [ ],
"numSourceExceptions" : 0,
"latestSourceExceptions" : [ ],
"numWritten" : 0,
"lastReceivedTime" : 0,
"workerId" : "c-standalone-fw-localhost-8080"
}
} ]
}
```
If so,this problem with the content of the pulsar-io module.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source submitted with invalid configuration stucks without consuming from Kafka topics
ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source
submitted with invalid configuration stucks without consuming from Kafka topics
URL: https://github.com/apache/pulsar/issues/3294#issuecomment-453418838
Use 'source create' command we validate the parameter key is included, but
it does not verify that the value of the key is correct, so create
sucessflully. If value is error, should we return the create source failure?
kafka.yaml info:
```file
configs:
bootstrapServers: "xxx"
groupId: "pulsarUTset"
topic: "test_kafka_topic"
fetchMinBytes: 1
autoCommitEnabled: "true"
autoCommitIntervalMs: 1000
sessionTimeoutMs: 3000
```
or
```file
configs:
```
command like :
```shell
./bin/pulsar-admin source create --source-config-file
~/code/pulsar/pulsar_ms/file/kafka.yaml -t kafka --destination-topic-name
dest_topic --name kafka_fn
"Created successfully"
```
when I getstatus return info:
```shell
$ ./bin/pulsar-admin source getstatus --name kafka_fn
{
"numInstances" : 1,
"numRunning" : 1,
"instances" : [ {
"instanceId" : 0,
"status" : {
"running" : true,
"error" : "",
"numRestarts" : 0,
"numReceivedFromSource" : 0,
"numSystemExceptions" : 0,
"latestSystemExceptions" : [ ],
"numSourceExceptions" : 0,
"latestSourceExceptions" : [ ],
"numWritten" : 0,
"lastReceivedTime" : 0,
"workerId" : "c-standalone-fw-localhost-8080"
}
} ]
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ambition119 commented on issue #3294: [IO] [Kafka] a Kafka source submitted with invalid configuration stucks without consuming from Kafka topics
ambition119 commented on issue #3294: [IO] [Kafka] a Kafka source submitted
with invalid configuration stucks without consuming from Kafka topics
URL: https://github.com/apache/pulsar/issues/3294#issuecomment-453418838
Use 'source create' command we validate verify that the parameter key is
included, but it does not verify that the value of the key is correct, so
create sucessflully. If value is error, should we return the create source
failure?
kafka.yaml info:
```file
configs:
bootstrapServers: "xxx"
groupId: "pulsarUTset"
topic: "test_kafka_topic"
fetchMinBytes: 1
autoCommitEnabled: "true"
autoCommitIntervalMs: 1000
sessionTimeoutMs: 3000
```
command like :
```shell
./bin/pulsar-admin source create --source-config-file
~/code/pulsar/pulsar_ms/file/kafka.yaml -t kafka --destination-topic-name
dest_topic --name kafka_fn
"Created successfully"
```
when I getstatus return info:
```shell
$ ./bin/pulsar-admin source getstatus --name kafka_fn
{
"numInstances" : 1,
"numRunning" : 1,
"instances" : [ {
"instanceId" : 0,
"status" : {
"running" : true,
"error" : "",
"numRestarts" : 0,
"numReceivedFromSource" : 0,
"numSystemExceptions" : 0,
"latestSystemExceptions" : [ ],
"numSourceExceptions" : 0,
"latestSourceExceptions" : [ ],
"numWritten" : 0,
"lastReceivedTime" : 0,
"workerId" : "c-standalone-fw-localhost-8080"
}
} ]
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source submitted with invalid configuration stucks without consuming from Kafka topics
ambition119 edited a comment on issue #3294: [IO] [Kafka] a Kafka source
submitted with invalid configuration stucks without consuming from Kafka topics
URL: https://github.com/apache/pulsar/issues/3294#issuecomment-453418838
Use 'source create' command we validate the parameter key is included, but
it does not verify that the value of the key is correct, so create
sucessflully. If value is error, should we return the create source failure?
kafka.yaml info:
```file
configs:
bootstrapServers: "xxx"
groupId: "pulsarUTset"
topic: "test_kafka_topic"
fetchMinBytes: 1
autoCommitEnabled: "true"
autoCommitIntervalMs: 1000
sessionTimeoutMs: 3000
```
command like :
```shell
./bin/pulsar-admin source create --source-config-file
~/code/pulsar/pulsar_ms/file/kafka.yaml -t kafka --destination-topic-name
dest_topic --name kafka_fn
"Created successfully"
```
when I getstatus return info:
```shell
$ ./bin/pulsar-admin source getstatus --name kafka_fn
{
"numInstances" : 1,
"numRunning" : 1,
"instances" : [ {
"instanceId" : 0,
"status" : {
"running" : true,
"error" : "",
"numRestarts" : 0,
"numReceivedFromSource" : 0,
"numSystemExceptions" : 0,
"latestSystemExceptions" : [ ],
"numSourceExceptions" : 0,
"latestSourceExceptions" : [ ],
"numWritten" : 0,
"lastReceivedTime" : 0,
"workerId" : "c-standalone-fw-localhost-8080"
}
} ]
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] JesusIslam commented on issue #3359: Failed to build Golang client.
JesusIslam commented on issue #3359: Failed to build Golang client. URL: https://github.com/apache/pulsar/issues/3359#issuecomment-453418589 Turned out I missed this explanation in [this page.](https://pulsar.apache.org/docs/latest/clients/go/) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] JesusIslam edited a comment on issue #3359: Failed to build Golang client.
JesusIslam edited a comment on issue #3359: Failed to build Golang client. URL: https://github.com/apache/pulsar/issues/3359#issuecomment-453418589 Turned out I missed this explanation in [this page.](https://pulsar.apache.org/docs/latest/clients/go/) > Pulsar Go client library is based on the C++ client library. Follow the instructions for C++ library for installing the binaries through RPM, Deb or Homebrew packages. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] JesusIslam closed issue #3359: Failed to build Golang client.
JesusIslam closed issue #3359: Failed to build Golang client. URL: https://github.com/apache/pulsar/issues/3359 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
