[GitHub] [pulsar] merlimat commented on a change in pull request #3752: revise the schema default type not null
merlimat commented on a change in pull request #3752: revise the schema default
type not null
URL: https://github.com/apache/pulsar/pull/3752#discussion_r264982965
##
File path:
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
##
@@ -167,25 +168,47 @@ default T decode(byte[] bytes, byte[] schemaVersion) {
}
/**
- * Create a Avro schema type by extracting the fields of the specified
class.
+ * Create a Avro schema type by default configuration of the class
*
* @param clazz the POJO class to be used to extract the Avro schema
* @return a Schema instance
*/
static Schema AVRO(Class clazz) {
-return DefaultImplementation.newAvroSchema(clazz);
+return DefaultImplementation.newAvroSchema(new
SchemaDefinition<>(clazz));
}
/**
- * Create a JSON schema type by extracting the fields of the specified
class.
+ * Create a Avro schema type with schema definition
+ *
+ * @param schemaDefinition the definition of the schema
+ * @return a Schema instance
+ */
+static Schema AVRO(SchemaDefinition schemaDefinition) {
Review comment:
> I don't think AvroDefinition is a good name. it is confusing - does it
mean it is for avro schema, or it is for generated the avro formatted schema. I
think SchemaDefinition is much better.
Sure that's probably not a good naming. Though I'd be careful in using
constructor approach since it makes it harder to extend, compared to using
builder pattern.
> if pojo class is omitted, the schema definition is generated from the json
string.
Ok that would make the `Producer` which should still be fine (in the
Avro case) since the avro schema is the one that is really deciding the
serialization.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on a change in pull request #3752: revise the schema default type not null
sijie commented on a change in pull request #3752: revise the schema default
type not null
URL: https://github.com/apache/pulsar/pull/3752#discussion_r264978138
##
File path:
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
##
@@ -167,25 +168,47 @@ default T decode(byte[] bytes, byte[] schemaVersion) {
}
/**
- * Create a Avro schema type by extracting the fields of the specified
class.
+ * Create a Avro schema type by default configuration of the class
*
* @param clazz the POJO class to be used to extract the Avro schema
* @return a Schema instance
*/
static Schema AVRO(Class clazz) {
-return DefaultImplementation.newAvroSchema(clazz);
+return DefaultImplementation.newAvroSchema(new
SchemaDefinition<>(clazz));
}
/**
- * Create a JSON schema type by extracting the fields of the specified
class.
+ * Create a Avro schema type with schema definition
+ *
+ * @param schemaDefinition the definition of the schema
+ * @return a Schema instance
+ */
+static Schema AVRO(SchemaDefinition schemaDefinition) {
Review comment:
I don't think `AvroDefinition` is a good name. it is confusing - does it
mean it is for avro schema, or it is for generated the avro formatted schema. I
think `SchemaDefinition` is much better.
If we agree on `SchemaDefinition`, pojo class is better to be part of it.
that means the schema definition is generated from pojo. if pojo class is
omitted, the schema definition is generated from the json string.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on a change in pull request #3752: revise the schema default type not null
sijie commented on a change in pull request #3752: revise the schema default
type not null
URL: https://github.com/apache/pulsar/pull/3752#discussion_r264977328
##
File path:
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
##
@@ -167,25 +168,47 @@ default T decode(byte[] bytes, byte[] schemaVersion) {
}
/**
- * Create a Avro schema type by extracting the fields of the specified
class.
+ * Create a Avro schema type by default configuration of the class
*
* @param clazz the POJO class to be used to extract the Avro schema
* @return a Schema instance
*/
static Schema AVRO(Class clazz) {
-return DefaultImplementation.newAvroSchema(clazz);
+return DefaultImplementation.newAvroSchema(new
SchemaDefinition<>(clazz));
}
/**
- * Create a JSON schema type by extracting the fields of the specified
class.
+ * Create a Avro schema type with schema definition
+ *
+ * @param schemaDefinition the definition of the schema
+ * @return a Schema instance
+ */
+static Schema AVRO(SchemaDefinition schemaDefinition) {
+return DefaultImplementation.newAvroSchema(schemaDefinition);
+}
+
+
+/**
+ * Create a JSON schema type by default configuration of the class
*
* @param clazz the POJO class to be used to extract the JSON schema
* @return a Schema instance
*/
static Schema JSON(Class clazz) {
-return DefaultImplementation.newJSONSchema(clazz);
+return DefaultImplementation.newJSONSchema(new
SchemaDefinition<>(clazz));
+}
+
+/**
+ * Create a JSON schema type with schema definition
+ *
+ * @param schemaDefinition the definition of the schema
+ * @return a Schema instance
+ */
+static Schema JSON(SchemaDefinition schemaDefinition) {
Review comment:
AllowNull probably doesn't make a lot sense to JSON. but having the ability
to create a schema definition is good for query engines using pulsar. so I
would prefer keeping it for consistency as AVRO, as it would allow people to
customize how schema is generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #3801: fix message_id_serialize to empty slice
merlimat commented on issue #3801: fix message_id_serialize to empty slice URL: https://github.com/apache/pulsar/pull/3801#issuecomment-472280030 run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #3752: revise the schema default type not null
merlimat commented on a change in pull request #3752: revise the schema default
type not null
URL: https://github.com/apache/pulsar/pull/3752#discussion_r264963480
##
File path:
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/SchemaDefinition.java
##
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.api.schema;
+
+import lombok.Data;
+import lombok.EqualsAndHashCode;
+import lombok.ToString;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * A schema definition
+ * {@link org.apache.pulsar.client.api.Schema} for the schema definition value.
+ */
+@Data
+@EqualsAndHashCode
+@ToString
+public class SchemaDefinition {
+
+/**
+ * the schema definition class
+ */
+private final Class clazz;
+/**
+ * The flag of schema type always allow null
+ */
+private boolean alwaysAllowNull = true;
+/**
+ * The schema info properties
+ */
+private Map properties = new HashMap<>();
+
+
+public SchemaDefinition(Class clazz) {
+
+properties.put("__alwaysAllowNull", "true");
Review comment:
`"__alwaysAllowNull"` is used multiple times and it would have to be set as
a constant.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #3752: revise the schema default type not null
merlimat commented on a change in pull request #3752: revise the schema default
type not null
URL: https://github.com/apache/pulsar/pull/3752#discussion_r264964644
##
File path:
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
##
@@ -222,9 +245,9 @@ default T decode(byte[] bytes, byte[] schemaVersion) {
/**
* Create a schema instance that automatically deserialize messages
* based on the current topic schema.
- *
+ *
Review comment:
Do not remove the `` as they're used in the generated javadoc.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #3752: revise the schema default type not null
merlimat commented on a change in pull request #3752: revise the schema default
type not null
URL: https://github.com/apache/pulsar/pull/3752#discussion_r264964829
##
File path:
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/SchemaDefinition.java
##
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.api.schema;
+
+import lombok.Data;
+import lombok.EqualsAndHashCode;
+import lombok.ToString;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * A schema definition
+ * {@link org.apache.pulsar.client.api.Schema} for the schema definition value.
+ */
+@Data
+@EqualsAndHashCode
+@ToString
+public class SchemaDefinition {
+
+/**
+ * the schema definition class
+ */
+private final Class clazz;
+/**
+ * The flag of schema type always allow null
+ */
+private boolean alwaysAllowNull = true;
+/**
+ * The schema info properties
+ */
+private Map properties = new HashMap<>();
+
+
+public SchemaDefinition(Class clazz) {
+
+properties.put("__alwaysAllowNull", "true");
+this.clazz = clazz;
+
+}
+
+/**
+ * Set schema whether always allow null or not
Review comment:
We should explain here what are the implications of using
allowNull=true/false in the Avro context and indicate what is the default
behavior.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #3752: revise the schema default type not null
merlimat commented on a change in pull request #3752: revise the schema default
type not null
URL: https://github.com/apache/pulsar/pull/3752#discussion_r264963389
##
File path:
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/SchemaDefinition.java
##
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.api.schema;
+
+import lombok.Data;
+import lombok.EqualsAndHashCode;
+import lombok.ToString;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * A schema definition
+ * {@link org.apache.pulsar.client.api.Schema} for the schema definition value.
+ */
+@Data
Review comment:
We generally avoid using Lombok in public API classes. Even if it the code
is generated at compile time, it will show up when a user look at the API code
in the IDE.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #3752: revise the schema default type not null
merlimat commented on a change in pull request #3752: revise the schema default
type not null
URL: https://github.com/apache/pulsar/pull/3752#discussion_r264962895
##
File path:
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
##
@@ -167,25 +168,47 @@ default T decode(byte[] bytes, byte[] schemaVersion) {
}
/**
- * Create a Avro schema type by extracting the fields of the specified
class.
+ * Create a Avro schema type by default configuration of the class
*
* @param clazz the POJO class to be used to extract the Avro schema
* @return a Schema instance
*/
static Schema AVRO(Class clazz) {
-return DefaultImplementation.newAvroSchema(clazz);
+return DefaultImplementation.newAvroSchema(new
SchemaDefinition<>(clazz));
}
/**
- * Create a JSON schema type by extracting the fields of the specified
class.
+ * Create a Avro schema type with schema definition
+ *
+ * @param schemaDefinition the definition of the schema
+ * @return a Schema instance
+ */
+static Schema AVRO(SchemaDefinition schemaDefinition) {
+return DefaultImplementation.newAvroSchema(schemaDefinition);
+}
+
+
+/**
+ * Create a JSON schema type by default configuration of the class
*
* @param clazz the POJO class to be used to extract the JSON schema
* @return a Schema instance
*/
static Schema JSON(Class clazz) {
-return DefaultImplementation.newJSONSchema(clazz);
+return DefaultImplementation.newJSONSchema(new
SchemaDefinition<>(clazz));
+}
+
+/**
+ * Create a JSON schema type with schema definition
+ *
+ * @param schemaDefinition the definition of the schema
+ * @return a Schema instance
+ */
+static Schema JSON(SchemaDefinition schemaDefinition) {
Review comment:
I think that JSON is a bit different compared with AVRO in that it does not
have a canonical schema representation. In AVRO, the schema def is the source
of truth, but for JSON the pojo is typically the source of truth (and people
use annotations to customize the specific fields).
For now I'd prefer to not add it. We can always add it later if there are
good reasons for it (but we'd not be able to take it out once it's in there).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #3752: revise the schema default type not null
merlimat commented on a change in pull request #3752: revise the schema default
type not null
URL: https://github.com/apache/pulsar/pull/3752#discussion_r264964562
##
File path:
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/Schema.java
##
@@ -167,25 +168,47 @@ default T decode(byte[] bytes, byte[] schemaVersion) {
}
/**
- * Create a Avro schema type by extracting the fields of the specified
class.
+ * Create a Avro schema type by default configuration of the class
*
* @param clazz the POJO class to be used to extract the Avro schema
* @return a Schema instance
*/
static Schema AVRO(Class clazz) {
-return DefaultImplementation.newAvroSchema(clazz);
+return DefaultImplementation.newAvroSchema(new
SchemaDefinition<>(clazz));
}
/**
- * Create a JSON schema type by extracting the fields of the specified
class.
+ * Create a Avro schema type with schema definition
+ *
+ * @param schemaDefinition the definition of the schema
+ * @return a Schema instance
+ */
+static Schema AVRO(SchemaDefinition schemaDefinition) {
Review comment:
Instead of having the `SchemaDefinition`, what about leaving the pojo and
adding the options like:
```java
Schema.AVRO(MyObject.class,
AvroDefinition.builder().allowNull(false).build());
```
This could be extended in #3766 with:
```java
Schema.AVRO(MyObject.class, AvroDefinition.fromSchema("...json
definition..."));
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] merlimat commented on a change in pull request #3752: revise the schema default type not null
merlimat commented on a change in pull request #3752: revise the schema default
type not null
URL: https://github.com/apache/pulsar/pull/3752#discussion_r264963567
##
File path:
pulsar-client-api/src/main/java/org/apache/pulsar/client/api/schema/SchemaDefinition.java
##
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.client.api.schema;
+
+import lombok.Data;
+import lombok.EqualsAndHashCode;
+import lombok.ToString;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * A schema definition
+ * {@link org.apache.pulsar.client.api.Schema} for the schema definition value.
+ */
+@Data
+@EqualsAndHashCode
+@ToString
+public class SchemaDefinition {
+
+/**
+ * the schema definition class
+ */
+private final Class clazz;
+/**
+ * The flag of schema type always allow null
+ */
+private boolean alwaysAllowNull = true;
+/**
+ * The schema info properties
+ */
+private Map properties = new HashMap<>();
+
+
+public SchemaDefinition(Class clazz) {
+
+properties.put("__alwaysAllowNull", "true");
Review comment:
In any case, I don't understand why we need the property in the map when we
already have the boolean flag.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] wolfstudy commented on issue #3767: [go functions] support go pulsar function
wolfstudy commented on issue #3767: [go functions] support go pulsar function URL: https://github.com/apache/pulsar/issues/3767#issuecomment-472257297 @merlimat Thanks for your reply, I have sent the corresponding email to dev@ list. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar.wiki] branch master updated: Updated PIP 30: change authentication provider API to support mutual authentication (markdown)
This is an automated email from the ASF dual-hosted git repository. zhaijia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.wiki.git The following commit(s) were added to refs/heads/master by this push: new 490459f Updated PIP 30: change authentication provider API to support mutual authentication (markdown) 490459f is described below commit 490459f66b78c93793c63f6008413fc04d25ad9c Author: Jia Zhai AuthorDate: Wed Mar 13 10:19:49 2019 +0800 Updated PIP 30: change authentication provider API to support mutual authentication (markdown) --- ...ge-authentication-provider-API-to-support-mutual-authentication.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/PIP-30:-change-authentication-provider-API-to-support-mutual-authentication.md b/PIP-30:-change-authentication-provider-API-to-support-mutual-authentication.md index 8a4c264..9634530 100644 --- a/PIP-30:-change-authentication-provider-API-to-support-mutual-authentication.md +++ b/PIP-30:-change-authentication-provider-API-to-support-mutual-authentication.md @@ -1,9 +1,9 @@ -* **Status**: Discussing +* **Status**: Implemented * **Author**: Jia Zhai * **Pull Request**: https://github.com/apache/pulsar/pull/3677 * **Mailing List discussion**: -* **Release**: +* **Release**: 2.4.0 ## Motivation
[pulsar.wiki] branch master updated: Updated PIP 30: change authentication provider API to support mutual authentication (markdown)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.wiki.git
The following commit(s) were added to refs/heads/master by this push:
new 0638259 Updated PIP 30: change authentication provider API to support
mutual authentication (markdown)
0638259 is described below
commit 06382598a27ed8b519ba357bba03be41bc25ba18
Author: Jia Zhai
AuthorDate: Wed Mar 13 10:18:39 2019 +0800
Updated PIP 30: change authentication provider API to support mutual
authentication (markdown)
---
...rovider-API-to-support-mutual-authentication.md | 42 ++
1 file changed, 18 insertions(+), 24 deletions(-)
diff --git
a/PIP-30:-change-authentication-provider-API-to-support-mutual-authentication.md
b/PIP-30:-change-authentication-provider-API-to-support-mutual-authentication.md
index df2a9b0..8a4c264 100644
---
a/PIP-30:-change-authentication-provider-API-to-support-mutual-authentication.md
+++
b/PIP-30:-change-authentication-provider-API-to-support-mutual-authentication.md
@@ -16,20 +16,25 @@ So this PIP is try to discuss the interface changes to
support mutual authentica
In Pulsar, authentication is happened when a new connection is creating
between client and broker. When connecting, Client sends authentication data to
Broker by `CommandConnect`, and Broker do the authentication and once success
send command `CommandConnected` back to client.
-In PulsarApi.proto,
[CommandConnect](https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/proto/PulsarApi.proto#L173),
it contains `auth_method_name` and `auth_data` fields. But broker no need to
send auth data to client, so CommandConnected not contains auth data.
+In PulsarApi.proto,
[CommandConnect](https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/proto/PulsarApi.proto#L173),
it contains `auth_method_name` and `auth_data` fields. But broker no need to
send auth data to client, so add new command CommandAuthResponse and
CommandAuthChallenge to carry the data between client and broker.
```
-message CommandConnect {
- required string client_version = 1;
- optional AuthMethod auth_method = 2; // Deprecated. Use
"auth_method_name" instead.
- optional string auth_method_name = 5;
- optional bytes auth_data = 3;
- …
+message CommandAuthResponse {
+ optional string client_version = 1;
+ optional AuthData response = 2;
+ optional int32 protocol_version = 3 [default = 0];
}
-message CommandConnected {
- required string server_version = 1;
- optional int32 protocol_version = 2 [default = 0];
+message CommandAuthChallenge {
+ optional string server_version = 1;
+ optional AuthData challenge = 2;
+ optional int32 protocol_version = 3 [default = 0];
+}
+
+// To support mutual authentication type, such as Sasl, reuse this command to
mutual auth.
+message AuthData {
+ optional string auth_method_name = 1;
+ optional bytes auth_data = 2;
}
```
@@ -37,30 +42,19 @@ The propose is to reuse these 2 commands related to
connecting and auth, and als
A basic logic for the mutual authentication is like this :
-1, Client side newConnectCommand(authDataClient) and send to Broker;
+1, Client side newConnectCommand(init auth) and send to Broker;
2, Broker side handleConnect(authDataClient), do the auth in Broker side, and
get authDataBroker.
- If auth is complete Broker.newConnected(), finish the auth, and send command
back to Client.
-- If auth is not complete, Broker.newConnecting(authDataBroker) and send
command back to Client.
+- If auth is not complete, Broker.newAuthChallenge(authDataBroker) and send
command back to Client.
3, Client side
- If received Connected command, complete the auth, and connection
established.
-- If received Connecting command, do the auth with authDataBroker, and get
authDataClient, then send connect command back to Broker. Broker will repeat
the process of step 2 until auth complete.
+- If received AuthChallenge command, do the auth with authDataBroker, and get
authDataClient, then send AuthResponse back to Broker. Broker will repeat the
process of step 2 until auth complete.
## Changes
### Proto
-In PulsarApi.proto,
[CommandConnected](https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/proto/PulsarApi.proto#L197)
need to add auth data fields. So Broker could reuse this command to send auth
data back to Client.
-
-```
-message CommandConnected {
- required string server_version = 1;
- optional int32 protocol_version = 2 [default = 0];
- // To support mutual authentication type, such as Sasl, reuse this
command to do mutual auth.
- optional string auth_method_name = 3;
- optional bytes auth_data = 4;
-}
-```
### API changes
[pulsar] branch master updated: PIP-30: interface for mutual authentication (#3677)
This is an automated email from the ASF dual-hosted git repository.
zhaijia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 09e3ed8 PIP-30: interface for mutual authentication (#3677)
09e3ed8 is described below
commit 09e3ed8aa15579d2fb265ce039b8f8bdb5b9f59f
Author: Jia Zhai
AuthorDate: Wed Mar 13 10:14:17 2019 +0800
PIP-30: interface for mutual authentication (#3677)
This is to implement the mutual auth api discussed in "PIP-30: change
authentication provider API to support mutual authentication"
Mainly provide 2 new command CommandAuthResponse and CommandAuthChallenge
in proto, to support it.
---
.../authentication/AuthenticationDataSource.java | 28 +-
.../authentication/AuthenticationProvider.java | 23 +-
.../authentication/AuthenticationService.java | 13 +
.../broker/authentication/AuthenticationState.java | 53 +
.../OneStageAuthenticationState.java | 68 +
.../apache/pulsar/broker/service/ServerCnx.java| 160 +-
.../pulsar/broker/service/ServerCnxTest.java | 27 +-
.../client/api/MutualAuthenticationTest.java | 240 +++
.../apache/pulsar/client/api/Authentication.java | 20 +-
.../client/api/AuthenticationDataProvider.java | 32 +-
.../org/apache/pulsar/common/api/AuthData.java | 34 +
pulsar-client-cpp/lib/Commands.cc |6 +
.../org/apache/pulsar/client/impl/ClientCnx.java | 61 +-
.../org/apache/pulsar/common/api/Commands.java | 82 +
.../apache/pulsar/common/api/PulsarDecoder.java| 22 +
.../apache/pulsar/common/api/proto/PulsarApi.java | 1590
pulsar-common/src/main/proto/PulsarApi.proto | 26 +
17 files changed, 2411 insertions(+), 74 deletions(-)
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/AuthenticationDataSource.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/AuthenticationDataSource.java
index 9fc6cbe..fcc6dda 100644
---
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/AuthenticationDataSource.java
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/AuthenticationDataSource.java
@@ -18,8 +18,10 @@
*/
package org.apache.pulsar.broker.authentication;
+import java.io.IOException;
import java.net.SocketAddress;
import java.security.cert.Certificate;
+import org.apache.pulsar.common.api.AuthData;
/**
* Interface for accessing data which are used in variety of authentication
schemes on server side
@@ -31,7 +33,7 @@ public interface AuthenticationDataSource {
/**
* Check if data from TLS are available.
- *
+ *
* @return true if this authentication data contain data from TLS
*/
default boolean hasDataFromTls() {
@@ -39,7 +41,7 @@ public interface AuthenticationDataSource {
}
/**
- *
+ *
* @return a client certificate chain, or null if the data are not
available
*/
default Certificate[] getTlsCertificates() {
@@ -52,7 +54,7 @@ public interface AuthenticationDataSource {
/**
* Check if data from HTTP are available.
- *
+ *
* @return true if this authentication data contain data from HTTP
*/
default boolean hasDataFromHttp() {
@@ -60,7 +62,7 @@ public interface AuthenticationDataSource {
}
/**
- *
+ *
* @return a authentication scheme, or null if the request is
not be authenticated
*/
default String getHttpAuthType() {
@@ -68,7 +70,7 @@ public interface AuthenticationDataSource {
}
/**
- *
+ *
* @return a String containing the value of the specified
header, or null if the header
* does not exist.
*/
@@ -82,7 +84,7 @@ public interface AuthenticationDataSource {
/**
* Check if data from Pulsar protocol are available.
- *
+ *
* @return true if this authentication data contain data from Pulsar
protocol
*/
default boolean hasDataFromCommand() {
@@ -90,20 +92,28 @@ public interface AuthenticationDataSource {
}
/**
- *
+ *
* @return authentication data which is stored in a command
*/
default String getCommandData() {
return null;
}
+/**
+ * Evaluate and challenge the data that passed in, and return processed
data back.
+ * It is used for mutual authentication like SASL.
+ */
+default AuthData authenticate(AuthData data) throws IOException {
+throw new UnsupportedOperationException();
+}
+
/*
* Peer
*/
/**
* Check if data from peer are available.
- *
+ *
* @return true if this authentication data contain data from peer
*/
default boolean hasDataFromPeer() {
@@ -111,7 +121,7 @@ public
[GitHub] [pulsar] jiazhai merged pull request #3677: PIP-30: interface and mutual change authentication
jiazhai merged pull request #3677: PIP-30: interface and mutual change authentication URL: https://github.com/apache/pulsar/pull/3677 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated: Issue #3803: Make ManagedLedger read batch size configurable (#3808)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 0ce297c Issue #3803: Make ManagedLedger read batch size configurable
(#3808)
0ce297c is described below
commit 0ce297c5ffd6430b047728a779a0896d6b5fb7d9
Author: Sijie Guo
AuthorDate: Wed Mar 13 07:17:32 2019 +0800
Issue #3803: Make ManagedLedger read batch size configurable (#3808)
*Motivation*
Fixes #3803
Hardcoding is a very bad practice. It means we have no way to alter system
behavior
when production issues occur.
*Modifications*
introduce a few read batch related settings to make them configurable
---
conf/broker.conf | 11 ++
.../apache/pulsar/broker/ServiceConfiguration.java | 25 ++
.../PersistentDispatcherMultipleConsumers.java | 15 ++---
.../PersistentDispatcherSingleActiveConsumer.java | 9
.../service/persistent/PersistentReplicator.java | 13 +--
5 files changed, 54 insertions(+), 19 deletions(-)
diff --git a/conf/broker.conf b/conf/broker.conf
index 73e94cd..ba00cfa 100644
--- a/conf/broker.conf
+++ b/conf/broker.conf
@@ -186,6 +186,17 @@ dispatchThrottlingRatePerSubscribeInByte=0
# backlog.
dispatchThrottlingOnNonBacklogConsumerEnabled=true
+# Max number of entries to read from bookkeeper. By default it is 100 entries.
+dispatcherMaxReadBatchSize=100
+
+# Min number of entries to read from bookkeeper. By default it is 1 entries.
+# When there is an error occurred on reading entries from bookkeeper, the
broker
+# will backoff the batch size to this minimum number."
+dispatcherMinReadBatchSize=1
+
+# Max number of entries to dispatch for a shared subscription. By default it
is 20 entries.
+dispatcherMaxRoundRobinBatchSize=20
+
# Max number of concurrent lookup request broker allows to throttle heavy
incoming lookup traffic
maxConcurrentLookupRequest=5
diff --git
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
index 39c3e42..0fbe155 100644
---
a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
+++
b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -377,6 +377,31 @@ public class ServiceConfiguration implements
PulsarConfiguration {
+ " published messages and don't have backlog. This enables
dispatch-throttling for "
+ " non-backlog consumers as well.")
private boolean dispatchThrottlingOnNonBacklogConsumerEnabled = false;
+
+// <-- dispatcher read settings -->
+@FieldContext(
+dynamic = true,
+category = CATEGORY_SERVER,
+doc = "Max number of entries to read from bookkeeper. By default it is
100 entries."
+)
+private int dispatcherMaxReadBatchSize = 100;
+
+@FieldContext(
+dynamic = true,
+category = CATEGORY_SERVER,
+doc = "Min number of entries to read from bookkeeper. By default it is
1 entries."
++ "When there is an error occurred on reading entries from
bookkeeper, the broker"
++ " will backoff the batch size to this minimum number."
+)
+private int dispatcherMinReadBatchSize = 1;
+
+@FieldContext(
+dynamic = true,
+category = CATEGORY_SERVER,
+doc = "Max number of entries to dispatch for a shared subscription. By
default it is 20 entries."
+)
+private int dispatcherMaxRoundRobinBatchSize = 20;
+
@FieldContext(
dynamic = true,
category = CATEGORY_SERVER,
diff --git
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
index 02c3ea8..066e2b6 100644
---
a/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
+++
b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java
@@ -65,9 +65,6 @@ import com.google.common.collect.Lists;
*/
public class PersistentDispatcherMultipleConsumers extends
AbstractDispatcherMultipleConsumers implements Dispatcher, ReadEntriesCallback {
-private static final int MaxReadBatchSize = 100;
-private static final int MaxRoundRobinBatchSize = 20;
-
private final PersistentTopic topic;
private final ManagedCursor cursor;
@@ -105,7 +102,7 @@ public class PersistentDispatcherMultipleConsumers extends
AbstractDispatcherMu
this.redeliveryTracker =
this.serviceConfig.isSubscriptionRedeliveryTrackerEnabled()
?
[GitHub] [pulsar] merlimat merged pull request #3808: Issue #3803: Make ManagedLedger read batch size configurable
merlimat merged pull request #3808: Issue #3803: Make ManagedLedger read batch size configurable URL: https://github.com/apache/pulsar/pull/3808 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat closed issue #3803: Make ManagedLedger read batch size configurable
merlimat closed issue #3803: Make ManagedLedger read batch size configurable URL: https://github.com/apache/pulsar/issues/3803 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #3810: Allow AuthenticationProviderToken to be configured to accept different claims
merlimat commented on issue #3810: Allow AuthenticationProviderToken to be configured to accept different claims URL: https://github.com/apache/pulsar/issues/3810#issuecomment-472217241 @klevy-toast Seems to be a good enhancement to have. Do you plan to work on it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] srkukarni opened a new pull request #3811: enable stream storage by default in the bookkeeper config
srkukarni opened a new pull request #3811: enable stream storage by default in the bookkeeper config URL: https://github.com/apache/pulsar/pull/3811 ### Motivation Currently in the bookkeeper config, stream storage is not enabled. This means that to enable it, one has to modify two places(function worker as well as bookkeeper). This pr makes stream storage running by default. ### Modifications *Describe the modifications you've done.* ### Verifying this change - [ ] Make sure that the change passes the CI checks. *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (10MB)* - *Extended integration test for recovery after broker failure* ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API: (yes / no) - The schema: (yes / no / don't know) - The default values of configurations: (yes / no) - The wire protocol: (yes / no) - The rest endpoints: (yes / no) - The admin cli options: (yes / no) - Anything that affects deployment: (yes / no / don't know) ### Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) - If a feature is not applicable for documentation, explain why? - If a feature is not documented yet in this PR, please create a followup issue for adding the documentation This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] klevy-toast opened a new issue #3810: Allow AuthenticationProviderToken to be configured to accept different claims
klevy-toast opened a new issue #3810: Allow AuthenticationProviderToken to be configured to accept different claims URL: https://github.com/apache/pulsar/issues/3810 **Is your feature request related to a problem? Please describe.** I would like to use the AuthenticationProviderToken plugin with tokens that conform to the OAuth standard, but not necessarily the JWT standard, and identify the client by an arbitrary claim, not necessarily the "subject" claim (which is taken from line 109 of `AuthenticationProviderToken.java` : ` return jwt.getBody().getSubject();`). **Describe the solution you'd like** Add a configurable field to `AuthenticationProviderToken` (which defaults to "subject"), and use `jwt.getBody().get(tokenField);` instead of `.getSubject()`. This way, I can specify which claim to use. **Describe alternatives you've considered** I could set the `subject` claim in my token issuer, but I don't use that claim anywhere else in the system and would like to use the same token I could make my own AuthenticationProvider with the above changes, but I think that this feature may be useful to other users in similar situations, and the changes are very minor. **Additional context** N/a This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] aahmed-se commented on issue #3772: Force before and after methods to always run
aahmed-se commented on issue #3772: Force before and after methods to always run URL: https://github.com/apache/pulsar/pull/3772#issuecomment-472128828 We still need this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] srkukarni commented on issue #3772: Force before and after methods to always run
srkukarni commented on issue #3772: Force before and after methods to always run URL: https://github.com/apache/pulsar/pull/3772#issuecomment-472121559 @aahmed-se is this still active? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #3767: [go functions] support go pulsar function
merlimat commented on issue #3767: [go functions] support go pulsar function URL: https://github.com/apache/pulsar/issues/3767#issuecomment-472113327 @wolfstudy Added to the wiki at https://github.com/apache/pulsar/wiki/PIP-32%3A-Go-Function-API%2C-Instance-and-LocalRun Please send email to dev@ list to start discussion This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat closed issue #2662: Go functions support
merlimat closed issue #2662: Go functions support URL: https://github.com/apache/pulsar/issues/2662 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] merlimat commented on issue #2662: Go functions support
merlimat commented on issue #2662: Go functions support URL: https://github.com/apache/pulsar/issues/2662#issuecomment-472113074 tracked in #3767 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar.wiki] branch master updated: Updated Home (markdown)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar.wiki.git The following commit(s) were added to refs/heads/master by this push: new 6bd3aeb Updated Home (markdown) 6bd3aeb is described below commit 6bd3aeba97fb79d3df5aa8ec993365464d606eb2 Author: Matteo Merli AuthorDate: Tue Mar 12 10:55:16 2019 -0700 Updated Home (markdown) --- Home.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Home.md b/Home.md index f968189..9da1f82 100644 --- a/Home.md +++ b/Home.md @@ -8,6 +8,7 @@ We encourage to document any big change or feature or any addition to public use *Next Proposal Number: 32* ### Proposed +* [[PIP 32: Go Function API, Instance and LocalRun]] * [[PIP 31: Transaction Support]] * [[PIP 30: change authentication provider API to support mutual authentication]] * [[PIP 28: Pulsar Proxy Gateway Improvement]]
[pulsar.wiki] branch master updated: Created PIP-32: Go Function API, Instance and LocalRun (markdown)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.wiki.git
The following commit(s) were added to refs/heads/master by this push:
new 19f0148 Created PIP-32: Go Function API, Instance and LocalRun
(markdown)
19f0148 is described below
commit 19f014897ffb9bcd5dec26d66a24b4303f92fd11
Author: Matteo Merli
AuthorDate: Tue Mar 12 10:54:42 2019 -0700
Created PIP-32: Go Function API, Instance and LocalRun (markdown)
---
PIP-32:-Go-Function-API,-Instance-and-LocalRun.md | 164 ++
1 file changed, 164 insertions(+)
diff --git a/PIP-32:-Go-Function-API,-Instance-and-LocalRun.md
b/PIP-32:-Go-Function-API,-Instance-and-LocalRun.md
new file mode 100644
index 000..869b058
--- /dev/null
+++ b/PIP-32:-Go-Function-API,-Instance-and-LocalRun.md
@@ -0,0 +1,164 @@
+## Motivation
+
+The server and client sides of the Pulsar function use protobuf for
decoupling. In principle, the language supported by protobuf can be supported
by the pulsar function. This greatly simplifies our work on developing the go
function client, and we don't have to worry about what happens to the runtime
and the worker. We treat the instance of the go language to be implemented as
the client side, and the two interact through the protobuf protocol. We only
need to generate a pb file of the g [...]
+## Goals
+
+Implement a MVP (Minimum Viable Product) of Go Functions, which includes Go
Function API, and Go Instance Implementation. And be able to localrun a go
function.
+
+## None Goals
+
+Following areas are not considered for the first implementation of Go
Function. Each of them itself can be considered a separated project.
+
+- Metrics
+- Admin Operations
+- State
+- Schema
+- Secrets
+- Log Topic
+- thentication & Authorization
+## API
+
+### Context api
+
+```
+type FunctionContext struct {
+ InstanceConf *InstanceConf
+ UserConfigs map[string]interface{}
+ InputTopics []string
+ StartTimetime.Time
+ Producer pulsar.Producer
+}
+
+func NewFuncContext(conf *InstanceConf, client pulsar.Client, inTopics
[]string) *FunctionContext {}
+
+func (c *FunctionContext) GetInstanceID() int {
+ return c.InstanceConf.InstanceID
+}
+
+func (c *FunctionContext) GetInputTopics() []string {
+ return c.InputTopics
+}
+
+func (c *FunctionContext) GetOutputTopic() string {
+ return c.InstanceConf.FuncDetails.GetSink().Topic
+}
+
+func (c *FunctionContext) GetFuncTenant() string {
+ return c.InstanceConf.FuncDetails.Tenant
+}
+
+func (c *FunctionContext) GetFuncName() string {
+ return c.InstanceConf.FuncDetails.Name
+}
+
+func (c *FunctionContext) GetFuncNamespace() string {
+ return c.InstanceConf.FuncDetails.Namespace
+}
+
+func (c *FunctionContext) GetFuncID() string {
+ return c.InstanceConf.FuncID
+}
+
+func (c *FunctionContext) GetFuncVersion() string {
+ return c.InstanceConf.FuncVersion
+}
+
+func (c *FunctionContext) GetUserConfValue(key string) interface{} {
+ return c.UserConfigs[key]
+}
+
+func (c *FunctionContext) GetUserConfMap() map[string]interface{} {
+ return c.UserConfigs
+}
+
+type key struct{}
+
+var contextKey = {}
+
+func NewContext(parent context.Context, fc *FunctionContext) context.Context {
+ return context.WithValue(parent, contextKey, fc)
+}
+
+func FromContext(ctx context.Context) (*FunctionContext, bool) {
+ fc, ok := ctx.Value(contextKey).(*FunctionContext)
+ return fc, ok
+}
+```
+
+### Instance api
+
+
+We will introduce a new public API named `pulsarfunc.Start(funcName
interface{})`, Start receives the name of a function. (For details on why we
choose this approach, please checkout Section `proposed changes` and Section
`alternatives`.)
+
+```
+
+func Start(funcName interface{}) {}
+
+```
+ Rules:
+
+- function must be a function type
+- function may take between 0 and two arguments.
+- if there are two arguments, the first argument must satisfy the
"context.Context" interface.
+- function may return between 0 and two arguments.
+- if there are two return values, the second argument must be an error.
+- if there is one return value it must be an error.
+
+ Valid function signatures:
+
+- func ()
+- func () error
+- func (input) error
+- func () (output, error)
+- func (input) (output, error)
+- func (context.Context) error
+- func (context.Context, input) error
+- func (context.Context) (output, error)
+- func (context.Context, input) (output, error)
+
+
+## Example
+
+Below is an example on showing how to use go pulsar function API.
+
+```
+package main
+
+import (
+ "fmt"
+
+ pulsarfunc "github.com/apache/pulsar/pulsar-function-go"
+)
+
+func hello() {
+ fmt.Println("hello pulsar function")
+}
+
+func main(){
+ pulsarfunc.Start(hello)
+}
+```
+
+## Proposed Changes
+### Instance
+
+Different from python and java, Go
[pulsar-client-node] 06/07: set new version
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-client-node.git
commit 987578e694fceb6788f74224f90b6b8662a61f7a
Author: yfuruta
AuthorDate: Tue Mar 12 14:44:20 2019 +0900
set new version
---
package.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/package.json b/package.json
index cda8e3c..2455ce6 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "pulsar-client",
- "version": "2.3.0",
+ "version": "2.4.0-SNAPSHOT",
"description": "Pulsar Node.js client",
"main": "index.js",
"directories": {
[pulsar-client-node] 03/07: change 4 to 2 spaces and reformat
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-client-node.git
commit 196a69c30e238e1438f4964b6643bfc4b56ed903
Author: yfuruta
AuthorDate: Tue Mar 12 10:48:30 2019 +0900
change 4 to 2 spaces and reformat
---
.clang-format | 2 +-
src/Client.cc | 194 +++---
src/Client.h | 20 ++--
src/Consumer.cc | 243
src/Consumer.h| 26 +++---
src/ConsumerConfig.cc | 89 +-
src/ConsumerConfig.h | 24 ++---
src/Message.cc| 252 +-
src/Message.h | 60 ++--
src/MessageId.cc | 64 ++---
src/MessageId.h | 28 +++---
src/Producer.cc | 194 +++---
src/Producer.h| 22 ++---
src/ProducerConfig.cc | 181 ++--
src/ProducerConfig.h | 16 ++--
src/addon.cc | 10 +-
16 files changed, 707 insertions(+), 718 deletions(-)
diff --git a/.clang-format b/.clang-format
index cb40b50..f2d174b 100644
--- a/.clang-format
+++ b/.clang-format
@@ -17,7 +17,7 @@
BasedOnStyle: Google
-IndentWidth: 4
+IndentWidth: 2
ColumnLimit: 110
SortIncludes: false
BreakBeforeBraces: Custom
diff --git a/src/Client.cc b/src/Client.cc
index 97fc573..16fb13d 100644
--- a/src/Client.cc
+++ b/src/Client.cc
@@ -38,133 +38,133 @@ static const std::string CFG_STATS_INTERVAL =
"statsIntervalInSeconds";
Napi::FunctionReference Client::constructor;
Napi::Object Client::Init(Napi::Env env, Napi::Object exports) {
-Napi::HandleScope scope(env);
+ Napi::HandleScope scope(env);
-Napi::Function func = DefineClass(
-env, "Client",
-{InstanceMethod("createProducer", ::CreateProducer),
- InstanceMethod("subscribe", ::Subscribe),
InstanceMethod("close", ::Close)});
+ Napi::Function func =
+ DefineClass(env, "Client",
+ {InstanceMethod("createProducer", ::CreateProducer),
+ InstanceMethod("subscribe", ::Subscribe),
InstanceMethod("close", ::Close)});
-constructor = Napi::Persistent(func);
-constructor.SuppressDestruct();
+ constructor = Napi::Persistent(func);
+ constructor.SuppressDestruct();
-exports.Set("Client", func);
-return exports;
+ exports.Set("Client", func);
+ return exports;
}
Client::Client(const Napi::CallbackInfo ) :
Napi::ObjectWrap(info) {
-Napi::Env env = info.Env();
-Napi::HandleScope scope(env);
-Napi::Object clientConfig = info[0].As();
-
-if (!clientConfig.Has(CFG_SERVICE_URL) ||
!clientConfig.Get(CFG_SERVICE_URL).IsString()) {
-if (clientConfig.Get(CFG_SERVICE_URL).ToString().Utf8Value().empty()) {
-Napi::Error::New(env, "Service URL is required and must be
specified as a string")
-.ThrowAsJavaScriptException();
-return;
-}
+ Napi::Env env = info.Env();
+ Napi::HandleScope scope(env);
+ Napi::Object clientConfig = info[0].As();
+
+ if (!clientConfig.Has(CFG_SERVICE_URL) ||
!clientConfig.Get(CFG_SERVICE_URL).IsString()) {
+if (clientConfig.Get(CFG_SERVICE_URL).ToString().Utf8Value().empty()) {
+ Napi::Error::New(env, "Service URL is required and must be specified as
a string")
+ .ThrowAsJavaScriptException();
+ return;
}
-Napi::String serviceUrl = clientConfig.Get(CFG_SERVICE_URL).ToString();
+ }
+ Napi::String serviceUrl = clientConfig.Get(CFG_SERVICE_URL).ToString();
-pulsar_client_configuration_t *cClientConfig =
pulsar_client_configuration_create();
+ pulsar_client_configuration_t *cClientConfig =
pulsar_client_configuration_create();
-if (clientConfig.Has(CFG_OP_TIMEOUT) &&
clientConfig.Get(CFG_OP_TIMEOUT).IsNumber()) {
-int32_t operationTimeoutSeconds =
clientConfig.Get(CFG_OP_TIMEOUT).ToNumber().Int32Value();
-if (operationTimeoutSeconds > 0) {
-
pulsar_client_configuration_set_operation_timeout_seconds(cClientConfig,
operationTimeoutSeconds);
-}
+ if (clientConfig.Has(CFG_OP_TIMEOUT) &&
clientConfig.Get(CFG_OP_TIMEOUT).IsNumber()) {
+int32_t operationTimeoutSeconds =
clientConfig.Get(CFG_OP_TIMEOUT).ToNumber().Int32Value();
+if (operationTimeoutSeconds > 0) {
+ pulsar_client_configuration_set_operation_timeout_seconds(cClientConfig,
operationTimeoutSeconds);
}
+ }
-if (clientConfig.Has(CFG_IO_THREADS) &&
clientConfig.Get(CFG_IO_THREADS).IsNumber()) {
-int32_t ioThreads =
clientConfig.Get(CFG_IO_THREADS).ToNumber().Int32Value();
-if (ioThreads > 0) {
-pulsar_client_configuration_set_io_threads(cClientConfig,
ioThreads);
-}
+ if (clientConfig.Has(CFG_IO_THREADS) &&
clientConfig.Get(CFG_IO_THREADS).IsNumber()) {
+int32_t ioThreads =
[pulsar-client-node] 01/07: Initialize pulsar-client-node project
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar-client-node.git commit 060a66246a38a3d9e367f41ba44c4f80db534838 Author: Sijie Guo AuthorDate: Fri Mar 1 21:47:17 2019 +0800 Initialize pulsar-client-node project --- LICENSE | 305 ++ README.md | 1 + 2 files changed, 306 insertions(+) diff --git a/LICENSE b/LICENSE new file mode 100644 index 000..356931c --- /dev/null +++ b/LICENSE @@ -0,0 +1,305 @@ + + Apache License + Version 2.0, January 2004 +http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to
[pulsar-client-node] 05/07: add compatibility and change version
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-client-node.git
commit d4513f7b3e03df6cd185f3cec3f81a74a5d4fe25
Author: yfuruta
AuthorDate: Tue Mar 12 12:45:36 2019 +0900
add compatibility and change version
---
README.md| 4
package.json | 2 +-
2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/README.md b/README.md
index 90d856f..0418c32 100644
--- a/README.md
+++ b/README.md
@@ -23,6 +23,10 @@
The Pulsar Node.js client can be used to create Pulsar producers and consumers
in Node.js.
+## Compatibility
+
+This Node.js client is developed and tested using Apache Pulsar 2.3.0
+
## Requirements
Pulsar Node.js client library is based on the C++ client library. Follow the
instructions for
diff --git a/package.json b/package.json
index 33b1321..cda8e3c 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "pulsar-client",
- "version": "0.0.1",
+ "version": "2.3.0",
"description": "Pulsar Node.js client",
"main": "index.js",
"directories": {
[pulsar-client-node] branch master updated (060a662 -> 2f977b9)
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/pulsar-client-node.git. from 060a662 Initialize pulsar-client-node project new f866cd6 provide nodejs client new 196a69c change 4 to 2 spaces and reformat new 05c9e48 make config inline new d4513f7 add compatibility and change version new 987578e set new version new 2f977b9 Merge pull request #1 from k2la/provide_nodejs_client The 7 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .clang-format | 25 + .eslintrc.json|9 + .gitignore|3 + Gruntfile.js | 31 + README.md | 56 +- binding.gyp | 42 + examples/consumer.js | 45 + examples/producer.js | 49 + index.js | 28 + package-lock.json | 2762 + package.json | 46 + perf/perf_consumer.js | 103 ++ perf/perf_producer.js | 119 +++ src/Client.cc | 170 +++ src/Client.h | 41 + src/Consumer.cc | 183 src/Consumer.h| 44 + src/ConsumerConfig.cc | 105 ++ src/ConsumerConfig.h | 44 + src/Message.cc| 197 src/Message.h | 64 ++ src/MessageId.cc | 87 ++ src/MessageId.h | 46 + src/Producer.cc | 151 +++ src/Producer.h| 42 + src/ProducerConfig.cc | 158 +++ src/ProducerConfig.h | 38 + src/addon.cc | 35 + 28 files changed, 4722 insertions(+), 1 deletion(-) create mode 100644 .clang-format create mode 100644 .eslintrc.json create mode 100644 .gitignore create mode 100644 Gruntfile.js create mode 100644 binding.gyp create mode 100644 examples/consumer.js create mode 100644 examples/producer.js create mode 100644 index.js create mode 100644 package-lock.json create mode 100644 package.json create mode 100644 perf/perf_consumer.js create mode 100644 perf/perf_producer.js create mode 100644 src/Client.cc create mode 100644 src/Client.h create mode 100644 src/Consumer.cc create mode 100644 src/Consumer.h create mode 100644 src/ConsumerConfig.cc create mode 100644 src/ConsumerConfig.h create mode 100644 src/Message.cc create mode 100644 src/Message.h create mode 100644 src/MessageId.cc create mode 100644 src/MessageId.h create mode 100644 src/Producer.cc create mode 100644 src/Producer.h create mode 100644 src/ProducerConfig.cc create mode 100644 src/ProducerConfig.h create mode 100644 src/addon.cc
[pulsar-client-node] 07/07: Merge pull request #1 from k2la/provide_nodejs_client
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/pulsar-client-node.git commit 2f977b959c0b5013987a77b9a4ce6d3d79c2c14c Merge: 060a662 987578e Author: Matteo Merli AuthorDate: Tue Mar 12 09:34:05 2019 -0700 Merge pull request #1 from k2la/provide_nodejs_client Provide Node Client Library .clang-format | 25 + .eslintrc.json|9 + .gitignore|3 + Gruntfile.js | 31 + README.md | 56 +- binding.gyp | 42 + examples/consumer.js | 45 + examples/producer.js | 49 + index.js | 28 + package-lock.json | 2762 + package.json | 46 + perf/perf_consumer.js | 103 ++ perf/perf_producer.js | 119 +++ src/Client.cc | 170 +++ src/Client.h | 41 + src/Consumer.cc | 183 src/Consumer.h| 44 + src/ConsumerConfig.cc | 105 ++ src/ConsumerConfig.h | 44 + src/Message.cc| 197 src/Message.h | 64 ++ src/MessageId.cc | 87 ++ src/MessageId.h | 46 + src/Producer.cc | 151 +++ src/Producer.h| 42 + src/ProducerConfig.cc | 158 +++ src/ProducerConfig.h | 38 + src/addon.cc | 35 + 28 files changed, 4722 insertions(+), 1 deletion(-)
[pulsar-client-node] 04/07: make config inline
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar-client-node.git
commit 05c9e48f397522395e0411d3c28aad354ac05e9a
Author: yfuruta
AuthorDate: Tue Mar 12 10:55:19 2019 +0900
make config inline
---
examples/consumer.js | 10 --
examples/producer.js | 10 --
2 files changed, 8 insertions(+), 12 deletions(-)
diff --git a/examples/consumer.js b/examples/consumer.js
index 7092c6b..c754ac3 100644
--- a/examples/consumer.js
+++ b/examples/consumer.js
@@ -21,19 +21,17 @@ const Pulsar = require('../index.js');
(async () => {
// Create a client
- const clientConfig = {
+ const client = new Pulsar.Client({
serviceUrl: 'pulsar://localhost:6650',
operationTimeoutSeconds: 30,
- };
- const client = new Pulsar.Client(clientConfig);
+ });
// Create a consumer
- const consumerConfig = {
+ const consumer = await client.subscribe({
topic: 'persistent://public/default/my-topic',
subscription: 'sub1',
ackTimeoutMs: 1,
- };
- const consumer = await client.subscribe(consumerConfig);
+ });
// Receive messages
for (let i = 0; i < 10; i += 1) {
diff --git a/examples/producer.js b/examples/producer.js
index b1b6856..8730719 100644
--- a/examples/producer.js
+++ b/examples/producer.js
@@ -21,19 +21,17 @@ const Pulsar = require('../index.js');
(async () => {
// Create a client
- const clientConfig = {
+ const client = new Pulsar.Client({
serviceUrl: 'pulsar://localhost:6650',
operationTimeoutSeconds: 30,
- };
- const client = new Pulsar.Client(clientConfig);
+ });
// Create a producer
- const producerConfig = {
+ const producer = await client.createProducer({
topic: 'persistent://public/default/my-topic',
sendTimeoutMs: 3,
batchingEnabled: true,
- };
- const producer = await client.createProducer(producerConfig);
+ });
// Send messages
const results = [];
[GitHub] [pulsar] sijie commented on issue #3808: Issue #3803: Make ManagedLedger read batch size configurable
sijie commented on issue #3808: Issue #3803: Make ManagedLedger read batch size configurable URL: https://github.com/apache/pulsar/pull/3808#issuecomment-472078125 run cpp tests run integration tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] david-streamlio opened a new issue #3809: Source jars do not contain any source code
david-streamlio opened a new issue #3809: Source jars do not contain any source code URL: https://github.com/apache/pulsar/issues/3809 **Describe the bug** The source jar published to http://central.maven.org/maven2/org/apache/pulsar/pulsar-client-admin/2.3.0/pulsar-client-admin-2.3.0-sources.jar, does not contain any source code **To Reproduce** Steps to reproduce the behavior: 1. Go to 'http://central.maven.org/maven2/org/apache/pulsar/pulsar-client-admin/2.3.0/' 2. Click on 'pulsar-client-admin-2.3.0-sources.jar' to download the jar 3. List the jar contents, 'jar -tvf ~/Downloads/pulsar-client-admin-2.3.0-sources.jar` 4. See error jar -tvf ~/Downloads/pulsar-client-admin-2.3.0-sources.jar 126 Thu Feb 14 22:41:24 PST 2019 META-INF/MANIFEST.MF 0 Thu Feb 14 22:41:24 PST 2019 META-INF/ 0 Thu Feb 14 22:41:24 PST 2019 META-INF/maven/ 0 Thu Feb 14 22:41:24 PST 2019 META-INF/maven/org.apache.pulsar/ 0 Thu Feb 14 22:41:24 PST 2019 META-INF/maven/org.apache.pulsar/pulsar-client-admin/ 11358 Thu Feb 14 22:41:24 PST 2019 META-INF/LICENSE 21319 Thu Feb 14 22:41:24 PST 2019 META-INF/DEPENDENCIES 132 Thu Feb 14 22:12:12 PST 2019 META-INF/maven/org.apache.pulsar/pulsar-client-admin/pom.properties 172 Thu Feb 14 22:41:24 PST 2019 META-INF/NOTICE 10627 Thu Feb 14 22:06:46 PST 2019 META-INF/maven/org.apache.pulsar/pulsar-client-admin/pom.xml **Expected behavior** The jar file should contain the source code for the pulsar client This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] ivankelly commented on a change in pull request #3677: PIP-30: interface and mutual change authentication
ivankelly commented on a change in pull request #3677: PIP-30: interface and
mutual change authentication
URL: https://github.com/apache/pulsar/pull/3677#discussion_r264711983
##
File path:
pulsar-client-api/src/main/java/org/apache/pulsar/common/api/AuthData.java
##
@@ -20,8 +20,12 @@
import lombok.Data;
+import static java.nio.charset.StandardCharsets.UTF_8;
+
@Data(staticConstructor="of")
public final class AuthData {
+public static byte[] INIT_AUTH_DATA = "init".getBytes(UTF_8);
Review comment:
"init" is too generic here. It's not outside the realm of possibility that
some auth protocol would send "init" in as the challenge. Add something more
specific to pulsar. "PulsarAuthInit" or so
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie commented on issue #3804: ManagedLedger should streamline the read requests
sijie commented on issue #3804: ManagedLedger should streamline the read requests URL: https://github.com/apache/pulsar/issues/3804#issuecomment-472014740 I have seen in a production deployment the consumer can never catch up if bookie's avg read latency is 10+ms bad due to the disk and other workloads running on same machine. the problem can potentially be mitigated if we can have a proper readahead mechanism in managed like what dlog is doing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Tevic commented on issue #3801: fix message_id_serialize to empty slice
Tevic commented on issue #3801: fix message_id_serialize to empty slice URL: https://github.com/apache/pulsar/pull/3801#issuecomment-471981946 > @Tevic is it possible to add a simple unit test for this? Sure, i'm not familiar with c++ so i just add the golang test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] sijie merged pull request #3700: [pulsar-io] Add a Pulsar IO connector for Redis sink
sijie merged pull request #3700: [pulsar-io] Add a Pulsar IO connector for Redis sink URL: https://github.com/apache/pulsar/pull/3700 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch master updated: [pulsar-io] Add a Pulsar IO connector for Redis sink (#3700)
This is an automated email from the ASF dual-hosted git repository.
sijie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new c5e3baa [pulsar-io] Add a Pulsar IO connector for Redis sink (#3700)
c5e3baa is described below
commit c5e3baafdadd26790358a80d5cf375726a186409
Author: Fangbin Sun
AuthorDate: Tue Mar 12 19:17:58 2019 +0800
[pulsar-io] Add a Pulsar IO connector for Redis sink (#3700)
### Motivation
This PR provides a built-in Redis sink Connector, in order to cache
messages in Redis [key-value] pairs. This will effectively make Redis a caching
system, which other applications can access to get the latest value.
### Modifications
Add a new sub-module in the `pulsar-io` module.
### Verifying this change
This change can be verified as follows:
* deploy the redis sink connector with configuration file containing the
following fields:
```
configs:
redisHosts: "localhost:6379"
redisPassword: "redis@123"
redisDatabase: "1"
clientMode: "Standalone"
operationTimeout: "3000"
batchSize: "100"
```
* start a redis instance with auth
* send messages with `NotNull` key/value in the topic declared when
deploying the connector
* check in Redis if the message's key-value pairs have been stored in above
database
### Documentation
```
# Submit a Redis Sink
$ bin/pulsar-admin sink create --tenant public --namespace default --name
redis-test-sink --sink-type redis --sink-config-file examples/redis-sink.yaml
--inputs test_redis
# List Sink
$ bin/pulsar-admin sink list --tenant public --namespace default
# Get Sink Info
$ bin/pulsar-admin sink get --tenant public --namespace default --name
redis-test-sink
# Get Sink Status
$ bin/pulsar-admin sink status --tenant public --namespace default --name
redis-test-sink
# Delete the Redis Sink
$ bin/pulsar-admin sink delete --tenant public --namespace default --name
redis-test-sink
```
---
pulsar-io/pom.xml | 1 +
pulsar-io/redis/pom.xml| 100
.../pulsar/io/redis/RedisAbstractConfig.java | 124 +++
.../org/apache/pulsar/io/redis/RedisSession.java | 140 +
.../org/apache/pulsar/io/redis/sink/RedisSink.java | 173 +
.../pulsar/io/redis/sink/RedisSinkConfig.java | 84 ++
.../resources/META-INF/services/pulsar-io.yaml | 21 +++
.../apache/pulsar/io/redis/EmbeddedRedisUtils.java | 63
.../pulsar/io/redis/sink/RedisSinkConfigTest.java | 149 ++
.../apache/pulsar/io/redis/sink/RedisSinkTest.java | 118 ++
pulsar-io/redis/src/test/resources/sinkConfig.yaml | 29
11 files changed, 1002 insertions(+)
diff --git a/pulsar-io/pom.xml b/pulsar-io/pom.xml
index ffd1e67..d45ba7b 100644
--- a/pulsar-io/pom.xml
+++ b/pulsar-io/pom.xml
@@ -53,6 +53,7 @@
hbase
mongo
flume
+redis
diff --git a/pulsar-io/redis/pom.xml b/pulsar-io/redis/pom.xml
new file mode 100644
index 000..62af469
--- /dev/null
+++ b/pulsar-io/redis/pom.xml
@@ -0,0 +1,100 @@
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+pulsar-io
+org.apache.pulsar
+2.4.0-SNAPSHOT
+
+
+pulsar-io-redis
+Pulsar IO :: Redis
+
+
+
+${project.parent.groupId}
+pulsar-io-core
+${project.parent.version}
+
+
+${project.groupId}
+pulsar-functions-instance
+${project.version}
+
+
+
+${project.groupId}
+pulsar-client-original
+${project.version}
+
+
+io.lettuce
+lettuce-core
+5.0.2.RELEASE
+
+
+com.google.guava
+guava
+
+
+com.fasterxml.jackson.core
+jackson-databind
+
+
+com.fasterxml.jackson.dataformat
+jackson-dataformat-yaml
+
+
+org.apache.commons
+commons-lang3
+3.4
+
+
+commons-collections
+commons-collections
+3.2.2
+
+
+${project.parent.groupId}
+buildtools
+${project.parent.version}
+test
+
+
+com.github.kstyrc
+embedded-redis
+0.6
+test
+
+
+
+
+
+
+
[GitHub] [pulsar] sijie opened a new pull request #3808: Issue #3803: Make ManagedLedger read batch size configurable
sijie opened a new pull request #3808: Issue #3803: Make ManagedLedger read batch size configurable URL: https://github.com/apache/pulsar/pull/3808 *Motivation* Fixes #3803 Hardcoding is a very bad practice. It means we have no way to alter system behavior when production issues occur. *Modifications* introduce a few read batch related settings to make them configurable ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): NO - The public API: NO - The schema: NO - The default values of configurations: NO - The wire protocol: NO - The rest endpoints: NO - The admin cli options: NO - Anything that affects deployment: NO ### Documentation - Does this pull request introduce a new feature? NO - If yes, how is the feature documented? (not applicable) - If a feature is not applicable for documentation, explain why? * it is self-documented in configuration file - If a feature is not documented yet in this PR, please create a followup issue for adding the documentation This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] ivankelly commented on a change in pull request #3735: Implementing authentication for Pulsar Functions
ivankelly commented on a change in pull request #3735: Implementing
authentication for Pulsar Functions
URL: https://github.com/apache/pulsar/pull/3735#discussion_r264621074
##
File path: pulsar-functions/proto/src/main/proto/Function.proto
##
@@ -141,6 +141,11 @@ message FunctionMetaData {
uint64 version = 3;
uint64 createTime = 4;
map instanceStates = 5;
+FunctionAuthenticationSpec functionAuthSpec = 6;
+}
+
+message FunctionAuthenticationSpec {
+bytes data = 1;
Review comment:
I think it should always include the credentials. If you want to make it
secure, make it secure using the caller of cacheAuthData, not within
cacheAuthData
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] jiazhai opened a new issue #3807: Ledger Offload issue S3 type setting
jiazhai opened a new issue #3807: Ledger Offload issue S3 type setting
URL: https://github.com/apache/pulsar/issues/3807
**Describe the bug**
User use setting :
```
managedLedgerOffloadDriver=S3
s3ManagedLedgerOffloadRegion=us
s3ManagedLedgerOffloadBucket=fio-dev-pulsar-topic-offload
s3ManagedLedgerOffloadServiceEndpoint=https://us-chhq.ceph.*
```
for ledgerOffload, and meet error when start broker:
```
16:53:31.930 [main] INFO
org.apache.bookkeeper.mledger.offload.OffloaderUtils - Found offloader
OffloaderDefinition(name=jcloud, description=JCloud based offloader
implementation,
offloaderFactoryClass=org.apache.bookkeeper.mledger.offload.jcloud.JCloudLedgerOffloaderFactory)
from /pulsar/./offloaders/tiered-storage-jcloud-2.2.1.nar
16:53:32.094 [load-factory-class
org.apache.bookkeeper.mledger.offload.jcloud.JCloudLedgerOffloaderFactory] INFO
org.apache.bookkeeper.mledger.offload.OffloaderUtils - Loading offloader
factory class
org.apache.bookkeeper.mledger.offload.jcloud.JCloudLedgerOffloaderFactory using
class loader
org.apache.pulsar.common.nar.NarClassLoader[/tmp/pulsar-nar/tiered-storage-jcloud-2.2.1.nar-unpacked]
16:53:32.133 [main] INFO
org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader
- Constructor offload driver: aws-s3, host: https://us-chhq.ceph.*,
container: fio-dev-pulsar-topic-offload, region: us
16:53:33.565 [main] INFO
org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader
- Connect to blobstore : driver: aws-s3, region: us, endpoint:
https://us-chhq.ceph.*
17:04:35.723 [bookkeeper-ml-scheduler-OrderedScheduler-4-0] INFO
org.apache.bookkeeper.mledger.offload.jcloud.impl.BlobStoreManagedLedgerOffloader
- Connect to blobstore : driver: aws-s3, region: , endpoint:
https://us-chhq.ceph.*
17:05:30.681 [offloader-OrderedScheduler-1-0] ERROR
org.jclouds.http.handlers.BackoffLimitedRetryHandler - Cannot retry after
server error, command has exceeded retry limit 100:
[method=org.jclouds.aws.s3.AWSS3Client.public abstract java.lang.String
org.jclouds.s3.S3Client.getBucketLocation(java.lang.String)[fio-dev-pulsar-topic-offload],
request=GET https://s3.amazonaws.com/fio-dev-pulsar-topic-offload?location
HTTP/1.1]
```
**Additional context**
In Offload, we leverage
[jclouds](https://jclouds.apache.org/reference/providers/#blobstore-providers)
to support different blob store provider, in which type "s3" and "aws-s3" are
different.
"aws-s3" is for service provided by Amazon S3;
while "s3" is for service that supported S3 API, but may not provided by
Amazon S3.
In this code, we need to keep "s3" instead of change it into "aws-s3"
```
public static BlobStoreManagedLedgerOffloader
create(TieredStorageConfigurationData conf,
Map
userMetadata,
OrderedScheduler
scheduler)
throws IOException {
String driver = conf.getManagedLedgerOffloadDriver();
if ("s3".equals(driver.toLowerCase())) {
driver = "aws-s3"; <
}
if (!driverSupported(driver)) {
throw new IOException(
"Not support this kind of driver as offload backend: " +
driver);
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] jiazhai opened a new issue #3806: NPE in PartitionedProducerImpl##ProducerStatsRecorderImpl
jiazhai opened a new issue #3806: NPE in
PartitionedProducerImpl##ProducerStatsRecorderImpl
URL: https://github.com/apache/pulsar/issues/3806
**Describe the bug**
meet NPE while call PartitionedProducerImpl.getStats()
**Screenshots**
**Additional context**
In PartitionedProducerImpl, the stats instance(ProducerStatsRecorderImpl) is
init like this.
```
stats = client.getConfiguration().getStatsIntervalSeconds() > 0 ? new
ProducerStatsRecorderImpl() : null;
```
while in ProducerStatsRecorderImpl(), all member is set to null;
```
public ProducerStatsRecorderImpl() {
numMsgsSent = null;
numBytesSent = null;
numSendFailed = null;
numAcksReceived = null;
totalMsgsSent = null;
totalBytesSent = null;
totalSendFailed = null;
totalAcksReceived = null;
ds = null;
}
```
so when call PartitionedProducerImpl.getStats(), it will meet NPE
```
@Override
public synchronized ProducerStatsRecorderImpl getStats() {
if (stats == null) {
return null;
}
stats.reset(); < ===
for (int i = 0; i < topicMetadata.numPartitions(); i++) {
stats.updateCumulativeStats(producers.get(i).getStats());
}
return stats;
}
void reset() {
numMsgsSent.reset(); < ===
numBytesSent.reset();
numSendFailed.reset();
numAcksReceived.reset();
totalMsgsSent.reset();
totalBytesSent.reset();
totalSendFailed.reset();
totalAcksReceived.reset();
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie opened a new issue #3805: Pulsar should use bookkeeper's default values
sijie opened a new issue #3805: Pulsar should use bookkeeper's default values URL: https://github.com/apache/pulsar/issues/3805 *Motivation* Currently Pulsar overrides some bookkeeper default values. For example, `maxPendingReadRequestsPerThread`. I would suggest making them using the default values provided by bookkeeper, rather than overriding the values in pulsar. This would make things much clearer and easier to maintain. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[pulsar] branch asf-site updated: Updated site at revision ff4db8d
This is an automated email from the ASF dual-hosted git repository. mmerli pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/pulsar.git The following commit(s) were added to refs/heads/asf-site by this push: new d528bd7 Updated site at revision ff4db8d d528bd7 is described below commit d528bd74ca00f42f34655a1cf67336626a29cc99 Author: jenkins AuthorDate: Tue Mar 12 08:57:46 2019 + Updated site at revision ff4db8d --- content/docs/en/next/adaptors-kafka.html | 2 +- content/docs/en/next/adaptors-kafka/index.html | 2 +- .../en/next/admin-api-non-persistent-topics.html | 2 +- .../admin-api-non-persistent-topics/index.html | 2 +- content/docs/fr/next/adaptors-kafka.html | 2 +- content/docs/fr/next/adaptors-kafka/index.html | 2 +- .../fr/next/admin-api-non-persistent-topics.html | 2 +- .../admin-api-non-persistent-topics/index.html | 2 +- content/docs/ja/next/adaptors-kafka.html | 2 +- content/docs/ja/next/adaptors-kafka/index.html | 2 +- .../ja/next/admin-api-non-persistent-topics.html | 2 +- .../admin-api-non-persistent-topics/index.html | 2 +- .../zh-CN/2.1.0-incubating/adaptors-spark.html | 2 +- .../2.1.0-incubating/adaptors-spark/index.html | 2 +- .../zh-CN/2.1.1-incubating/adaptors-spark.html | 2 +- .../2.1.1-incubating/adaptors-spark/index.html | 2 +- content/docs/zh-CN/2.2.0/adaptors-spark.html | 2 +- content/docs/zh-CN/2.2.0/adaptors-spark/index.html | 2 +- content/docs/zh-CN/2.2.1/adaptors-spark.html | 2 +- content/docs/zh-CN/2.2.1/adaptors-spark/index.html | 2 +- content/docs/zh-CN/adaptors-spark.html | 2 +- content/docs/zh-CN/adaptors-spark/index.html | 2 +- content/docs/zh-CN/next/adaptors-kafka.html| 2 +- content/docs/zh-CN/next/adaptors-kafka/index.html | 2 +- content/docs/zh-CN/next/adaptors-spark.html| 2 +- content/docs/zh-CN/next/adaptors-spark/index.html | 2 +- .../next/admin-api-non-persistent-topics.html | 2 +- .../admin-api-non-persistent-topics/index.html | 2 +- content/swagger/swagger.json | 48 +++--- 29 files changed, 52 insertions(+), 52 deletions(-) diff --git a/content/docs/en/next/adaptors-kafka.html b/content/docs/en/next/adaptors-kafka.html index a88c35b..abce83c 100644 --- a/content/docs/en/next/adaptors-kafka.html +++ b/content/docs/en/next/adaptors-kafka.html @@ -197,7 +197,7 @@ consumer.subscribe(Arrays.asList(topic)); request.timeout.msIgnored retriesIgnoredPulsar client retries with exponential backoff until the send timeout expires send.buffer.bytesIgnored -timeout.msIgnored +timeout.msYes value.serializerYes diff --git a/content/docs/en/next/adaptors-kafka/index.html b/content/docs/en/next/adaptors-kafka/index.html index a88c35b..abce83c 100644 --- a/content/docs/en/next/adaptors-kafka/index.html +++ b/content/docs/en/next/adaptors-kafka/index.html @@ -197,7 +197,7 @@ consumer.subscribe(Arrays.asList(topic)); request.timeout.msIgnored retriesIgnoredPulsar client retries with exponential backoff until the send timeout expires send.buffer.bytesIgnored -timeout.msIgnored +timeout.msYes value.serializerYes diff --git a/content/docs/en/next/admin-api-non-persistent-topics.html b/content/docs/en/next/admin-api-non-persistent-topics.html index 253673c..9c48f7e 100644 --- a/content/docs/en/next/admin-api-non-persistent-topics.html +++ b/content/docs/en/next/admin-api-non-persistent-topics.html @@ -96,7 +96,7 @@ persisting messages. type: This subscription type consumers: The list of connected consumers for this subscription consumerName: Internal identifier for this consumer, generated by the client library -availablePermits: The number of messages this consumer has space for in the client library's listen queue. A value of 0 means the client library's queue is full and receive() isn't being called. A nonzero value means this consumer is ready to be dispatched messages. +availablePermits: The number of messages this consumer has space for in the client library's listen queue. A value less than 1 means the client library's queue is full and receive() isn't being called. A non-negative value means this consumer is ready to be dispatched messages. replication: This section gives the stats for cross-colo replication of this topic connected: Whether the outbound replicator is connected inboundConnection: The IP and port of the broker in the remote cluster's publisher connection to this broker diff --git a/content/docs/en/next/admin-api-non-persistent-topics/index.html b/content/docs/en/next/admin-api-non-persistent-topics/index.html index 253673c..9c48f7e 100644 --- a/content/docs/en/next/admin-api-non-persistent-topics/index.html +++ b/content/docs/en/next/admin-api-non-persistent-topics/index.html @@ -96,7 +96,7 @@ persisting messages. type: This subscription type
[GitHub] [pulsar] jiazhai edited a comment on issue #3677: PIP-30: interface and mutual change authentication
jiazhai edited a comment on issue #3677: PIP-30: interface and mutual change authentication URL: https://github.com/apache/pulsar/pull/3677#issuecomment-471896878 run integration tests run java8 tests ``` org.apache.pulsar.broker.admin.PersistentTopicsTest.cleanup org.apache.pulsar.tests.integration.functions.PulsarFunctionsThreadTest.pulsar-test-suite ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #3677: PIP-30: interface and mutual change authentication
jiazhai commented on issue #3677: PIP-30: interface and mutual change authentication URL: https://github.com/apache/pulsar/pull/3677#issuecomment-471896878 run integration tests run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] murong00 commented on issue #3700: [pulsar-io] Add a Pulsar IO connector for Redis sink
murong00 commented on issue #3700: [pulsar-io] Add a Pulsar IO connector for Redis sink URL: https://github.com/apache/pulsar/pull/3700#issuecomment-471886455 run cpp tests run java8 tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy commented on issue #3767: [go functions] support go pulsar function
wolfstudy commented on issue #3767: [go functions] support go pulsar function URL: https://github.com/apache/pulsar/issues/3767#issuecomment-471884898 [PIP-32: Support Go Function - API, Instance and LocalRun](https://gist.github.com/wolfstudy/773871ee0d24643f8111ccfa055b2f26) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy commented on issue #3802: [go client] cannot use size (type _Ctype_int) as type _Ctype_ulong
wolfstudy commented on issue #3802: [go client] cannot use size (type _Ctype_int) as type _Ctype_ulong URL: https://github.com/apache/pulsar/issues/3802#issuecomment-471883117 At first, I think it is a problem that go is inconsistent with the version of cpp client. so, exec `go clean -modcache` , `go mod tidy` again. Unfortunately, I still encountered the same problem. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] wolfstudy opened a new issue #3802: [go client] cannot use size (type _Ctype_int) as type _Ctype_ulong
wolfstudy opened a new issue #3802: [go client] cannot use size (type _Ctype_int) as type _Ctype_ulong URL: https://github.com/apache/pulsar/issues/3802 **Describe the bug** ``` # github.com/apache/pulsar/pulsar-client-go/pulsar ../../pulsar/c_message.go:91:213: cannot use size (type _Ctype_int) as type _Ctype_ulong in argument to func literal ``` **To Reproduce** Steps to reproduce the behavior: ``` 1. git clone g...@github.com:apache/pulsar.git 2. cd apache/pulsar/pulsar-client-cpp 3. cmake . 4. make 5. make install 6. cd ../pulsar-client-go 7. go build -o main producer.go ``` **Expected behavior** build success **Desktop (please complete the following information):** - OS: MAC OS - go: go 1.11.4 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] Tevic edited a comment on issue #3177: MessageID always seems to serialize to empty slice in Go client Reader
Tevic edited a comment on issue #3177: MessageID always seems to serialize to empty slice in Go client Reader URL: https://github.com/apache/pulsar/issues/3177#issuecomment-471866729 I've figure out why and test success. #3801 But if use DeserializeMessageID get a new ID and use consumer.AckID it will panic, i don't know why. @sijie ``` Code cause panic: id := msg.ID().Serialize() idParsed := pulsar.DeserializeMessageID(id) err = consumer.AckID(idParsed) Panic info: fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x7fdbd4e1127f] runtime stack: runtime.throw(0x517df2, 0x2a) /home/tevic/Work/RunTime/Go/src/runtime/panic.go:617 +0x72 runtime.sigpanic() /home/tevic/Work/RunTime/Go/src/runtime/signal_unix.go:374 +0x4a9 goroutine 1 [syscall]: runtime.cgocall(0x4cceb0, 0xcdfbb0, 0x7e71c0) /home/tevic/Work/RunTime/Go/src/runtime/cgocall.go:128 +0x5b fp=0xcdfb80 sp=0xcdfb48 pc=0x4096bb github.com/apache/pulsar/pulsar-client-go/pulsar._Cfunc_pulsar_consumer_acknowledge_async_id(0x7fdba800e8e0, 0x1e61710, 0x0, 0x0) _cgo_gotypes.go:761 +0x45 fp=0xcdfbb0 sp=0xcdfb80 pc=0x4b83a5 github.com/apache/pulsar/pulsar-client-go/pulsar.(*consumer).AckID.func1(0xc9c080, 0x52ff40, 0xca8058) /home/tevic/Work/WorkSpace/Code/Go/src/github.com/apache/pulsar/pulsar-client-go/pulsar/c_consumer.go:243 +0xc2 fp=0xcdfbf8 sp=0xcdfbb0 pc=0x4c3c72 github.com/apache/pulsar/pulsar-client-go/pulsar.(*consumer).AckID(0xc9c080, 0x52ff40, 0xca8058, 0x1, 0x1) /home/tevic/Work/WorkSpace/Code/Go/src/github.com/apache/pulsar/pulsar-client-go/pulsar/c_consumer.go:243 +0x3f fp=0xcdfc20 sp=0xcdfbf8 pc=0x4bddcf main.main() /home/tevic/Work/WorkSpace/Code/GoProjs/HelloWorld/Pulsar/Recv.go:46 +0x7f3 fp=0xcdff98 sp=0xcdfc20 pc=0x4cc063 runtime.main() /home/tevic/Work/RunTime/Go/src/runtime/proc.go:200 +0x20c fp=0xcdffe0 sp=0xcdff98 pc=0x4319cc runtime.goexit() /home/tevic/Work/RunTime/Go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xcdffe8 sp=0xcdffe0 pc=0x45aa21 goroutine 17 [chan send, locked to thread]: github.com/apache/pulsar/pulsar-client-go/pulsar.pulsarMessageListenerProxy(0x7fdbc1ffa7e0, 0x7fdbad00, 0x1e636c0) /home/tevic/Work/WorkSpace/Code/Go/src/github.com/apache/pulsar/pulsar-client-go/pulsar/c_consumer.go:192 +0x11b github.com/apache/pulsar/pulsar-client-go/pulsar._cgoexpwrap_40dd75c63fd4_pulsarMessageListenerProxy(0x7fdbc1ffa7e0, 0x7fdbad00, 0x1e636c0) _cgo_gotypes.go:1867 +0x3f exit status 2 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [pulsar] ambition119 commented on a change in pull request #3695: Hbase Connector Integration Test
ambition119 commented on a change in pull request #3695: Hbase Connector
Integration Test
URL: https://github.com/apache/pulsar/pull/3695#discussion_r264533766
##
File path:
tests/integration/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarFunctionsTest.java
##
@@ -332,41 +347,73 @@ protected void getSinkStatus(String tenant, String
namespace, String sinkName) t
kvs.put(key, value);
producer.newMessage()
-.key(key)
-.value(obj)
-.send();
+ .key(key)
+ .value(obj)
+ .send();
}
return kvs;
}
+// This for HbaseSinkTester
+protected Map
produceSchemaMessagesToInputTopicForHbaseSink(String inputTopicName,
+
Schema schema) throws Exception {
+@Cleanup
+PulsarClient client = PulsarClient.builder()
+ .serviceUrl(pulsarCluster.getPlainTextServiceUrl())
+ .build();
+@Cleanup
+Producer producer = client.newProducer(schema)
+ .topic(inputTopicName)
+ .create();
+LinkedHashMap kvs = new LinkedHashMap<>();
+
+
+String key = "row_key";
+HbaseSinkTester.Foo obj = new HbaseSinkTester.Foo();
+obj.setRowKey("rowKey_value");
+obj.setName("name_value");
+obj.setAddress("address_value");
+obj.setAge(30);
+obj.setFlag(true);
+String value = new String(schema.encode(obj));
+
+kvs.put(key, value);
+producer.newMessage()
+ .key(key)
+ .value(obj)
+ .send();
+
+return kvs;
+}
+
protected void deleteSink(String tenant, String namespace, String
sinkName) throws Exception {
String[] commands = {
-PulsarCluster.ADMIN_SCRIPT,
-"sink",
-"delete",
-"--tenant", tenant,
-"--namespace", namespace,
-"--name", sinkName
+ PulsarCluster.ADMIN_SCRIPT,
+ "sink",
+ "delete",
+ "--tenant", tenant,
+ "--namespace", namespace,
+ "--name", sinkName
};
ContainerExecResult result =
pulsarCluster.getAnyWorker().execCmd(commands);
assertTrue(
-result.getStdout().contains("Deleted successfully"),
-result.getStdout()
+ result.getStdout().contains("Deleted successfully"),
+ result.getStdout()
);
assertTrue(
-result.getStderr().isEmpty(),
-result.getStderr()
+ result.getStderr().isEmpty(),
+ result.getStderr()
);
}
protected void getSinkInfoNotFound(String tenant, String namespace, String
sinkName) throws Exception {
String[] commands = {
-PulsarCluster.ADMIN_SCRIPT,
-"sink",
-"get",
-"--tenant", tenant,
-"--namespace", namespace,
-"--name", sinkName
+ PulsarCluster.ADMIN_SCRIPT,
Review comment:
> Seems the original code is 4 bytes aligned, but now it is not :)
> there are also several other places with this code alignment issue.
I format code lead, fix this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
