2019-03-12 09:55:50 UTC - Maarten Tielemans: Do the bookkeepers get used in any way during non-persistent publishing? CPU usage is ~0% (max ~10%) ---- 2019-03-12 10:07:08 UTC - jia zhai: Opened issue: <https://github.com/apache/pulsar/issues/3807> ---- 2019-03-12 11:35:58 UTC - bhagesharora: Hello everyone, I downloaded respected connectors(nar files) and placed into a connectors directory in pulsar directory in my linux environment we can see here bhagesharora93@pulsar-setup-2-3-0-n1-standalone:~/apache-pulsar-2.3.0/connectors$ ls pulsar-io-cassandra-2.3.0.nar pulsar-io-data-generator-2.3.0.nar But when I am testing using curl command it's comming empty bhagesharora93@pulsar-setup-2-3-0-n1-standalone:~$ curl -s <http://localhost:8080/admin/v2/functions/connectors> [] Here also it's coming empty bhagesharora93@pulsar-setup-2-3-0-n1-standalone:~/apache-pulsar-2.3.0$ ./bin/pulsar-admin source create --tenant test-tenant --namespace test-namespace --name generator --destinationTopicName generator_test --source-type data-generator Invalid source type 'data-generator' -- Available sources are: []
What would be the reason ?? ---- 2019-03-12 12:25:20 UTC - Igor Zubchenok: Hi guys You publish sources to <http://central.maven.org/maven2/org/apache/pulsar/pulsar-client-admin/2.3.0/> but there are no sources actually ---- 2019-03-12 13:33:00 UTC - Ryan Samo: Hey guys, I have an existing Pulsar cluster that works awesome so thanks for that! Recently I have been looking into Apache DistributedLog to use for app logging and noticed it was now baked into Bookkeeper. I decided to try it out by using the Zookeeper and Bookies I have backing Pulsar and it works fine, but as I dig deeper into DistributedLog, I find myself asking why would I use DistributedLog when Pulsar handles pub/sub but also readers? Do you all have any experience one way or the other? It just seems like it would be nice to leverage all that Pulsar offers i.e. namespacing, authentication, authorization, etc. rather than dealing with the DistributedLog API. Thoughts? ---- 2019-03-12 13:37:58 UTC - Sébastien de Melo: client.conf is needed, it doesn't work without it ---- 2019-03-12 13:40:41 UTC - Sébastien de Melo: Sorry, I forgot to answer yesterday. Our changes allow to automatically deploy a function when the brokers are ready. The initialization problem has been solved by making the brokers wait for zookeepers and bookkeepers to be ready before starting. I guess something created the namespace before the initialization could do it cleanly, causing the error above. ---- 2019-03-12 13:42:15 UTC - Ganga Lakshmanasamy: ---- 2019-03-12 13:42:33 UTC - Ganga Lakshmanasamy: @Ravi ---- 2019-03-12 13:44:51 UTC - Sébastien de Melo: Hello guys! We are using Pulsar functions and wondering whether they must be recreated when the brokers die. Where are they stored? Are they automatically restored when a new broker starts? And if all of them die and new ones are created? ---- 2019-03-12 13:46:01 UTC - Sijie Guo: I think you should use pulsar when possible. you might consider dlog as the same level as bookkeeper, since it is just a wrapper over bookkeeper’s api to simplify the usage. you might only consider dlog when if you want to build a different type of distributed systems that talks to bookkeeper directly :slightly_smiling_face: ---- 2019-03-12 13:47:14 UTC - Sijie Guo: > whether they must be recreated when the brokers die you dont. the function’s code is stored in bookkeeper, the metadata is stored in a system topic. the function instances are automatically scheduled when brokers/function workers are dead. ---- 2019-03-12 13:48:11 UTC - Sijie Guo: @Sébastien de Melo I think the helm scripts in pulsar repo let brokers wait for zookeeper and bookkeeper to be ready. no? ---- 2019-03-12 13:48:21 UTC - Sébastien de Melo: Ok, thanks! :+1: ---- 2019-03-12 13:49:44 UTC - Ryan Samo: Gotcha, thanks for all the tips @Sijie Guo . You’ve been a big help! ok_hand : Sijie Guo ---- 2019-03-12 13:51:33 UTC - Sébastien de Melo: @Sijie Guo Only zookeeper (unless I'm wrong) <https://github.com/apache/pulsar/blob/master/deployment/kubernetes/helm/pulsar/templates/broker-deployment.yaml> There is a single init container, named wait-zookeeper-ready. ---- 2019-03-12 13:52:16 UTC - Sébastien de Melo: I have added: - name: wait-bookkeeper-ready image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" imagePullPolicy: {{ .Values.image.pullPolicy }} command: ["sh", "-c"] args: - >- until nslookup {{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}-{{ add (.Values.bookkeeper.replicaCount | int) -1 }}.{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}.{{ .Values.namespace }}; do sleep 3; done; ---- 2019-03-12 14:22:29 UTC - Matteo Merli: Non-persistent topic wouldn’t use bookies at all, data is just in brokers memory ---- 2019-03-12 16:01:00 UTC - Sanjeev Kulkarni: @bhagesharora did you start the broker after the connectors were placed in the connectors directory? ---- 2019-03-12 16:01:05 UTC - Sanjeev Kulkarni: also is tjis standalone? ---- 2019-03-12 16:02:36 UTC - Sanjeev Kulkarni: @Ganga Lakshmanasamy how is the network setup between the docker running the client and the pulsar service? Is the pulsar service also running in the same docker? ---- 2019-03-12 16:15:52 UTC - Arivoli Tirouvingadame: @Arivoli Tirouvingadame has joined the channel ---- 2019-03-12 16:25:49 UTC - David Kjerrumgaard: @Igor Zubchenok Thanks for letting us know. I will file a ticket to get this corrected. ---- 2019-03-12 16:28:43 UTC - Sijie Guo: pulsar-client-admin is a shaded package. so the sources and javadoc will not be available in the moment. ---- 2019-03-12 16:29:06 UTC - Sijie Guo: you might consider pulsar-client-admin-original for sources and javadoc. ---- 2019-03-12 16:31:11 UTC - David Kjerrumgaard: Filed issue. <https://github.com/apache/pulsar/issues/3809> ---- 2019-03-12 17:47:39 UTC - Alexandre DUVAL: ```Unable to authenticate: Failed to authentication token: The parsed JWT indicates it was signed with the HS256 signature algorithm, but the specified signing key of type sun.security.rsa.RSAPublicKeyImpl may not be used to validate HS256 signatures. Because the specified signing key reflects a specific and expected algorithm, and the JWT does not reflect this algorithm, it is likely that the JWT was not expected and therefore should not be trusted. Another possibility is that the parser was configured with the incorrect signing key, but this cannot be assumed for security reasons.``` Hi, i got this, but I'm using RS256 (pair keys for JWT). I only have errors on proxy nodes but the request works and it's a WARN log level. I don't understand why, I was using secret key and not pair keys before, does proxy has cache for this or something to clean? ---- 2019-03-12 18:26:19 UTC - Guy Feldman: cool ---- 2019-03-12 18:26:34 UTC - Guy Feldman: are the functions working well? ---- 2019-03-12 18:41:13 UTC - Chris DiGiovanni: It doesn't seem that there is a provider for s3 in that nar file... ---- 2019-03-12 18:42:25 UTC - Chris DiGiovanni: <https://gist.github.com/digi691/226ab832dee8d2b4994cb1ab8575cbd1> ---- 2019-03-12 18:52:46 UTC - Shivji Kumar Jha: @Matteo Merli Hey I have a topic where an avro schema was added using the rest API. I now want to create a producer for the topic using the java client client.newProducer() This Fails because this is assumed as Schema.BYTES and I have this exception : Exception in thread "main" org.apache.pulsar.client.api.PulsarClientException$IncompatibleSchemaException: org.apache.pulsar.broker.service.schema.IncompatibleSchemaException: Producers cannot connect without a schema to topics with a schema Now if I instead say .newProducer(Schema.AVRO(<java-class-representing-the-schma.class>)) Exception in thread "main" org.apache.pulsar.client.api.PulsarClientException$IncompatibleSchemaException: org.apache.pulsar.broker.service.schema.IncompatibleSchemaException: Incompatible schema used ---- 2019-03-12 18:53:32 UTC - Shivji Kumar Jha: Is there a way to say use the AVRO schema thats there on topic and create the producer? And then producer.send(object) ? ---- 2019-03-12 19:59:28 UTC - Prasanna Ram Venkatachalam: @Prasanna Ram Venkatachalam has joined the channel ---- 2019-03-12 21:31:18 UTC - Sam Leung: Hi! I am looking at <https://pulsar.apache.org/docs/en/security-extending/> and there are no instructions of how to build and include my own `AuthenticationProvider`. Are we just expected to compile a custom build of Pulsar if we want to extend auth? ---- 2019-03-12 21:34:27 UTC - Ali Ahmed: @sam.huang you can compile a compliant jar and add it to the class path. ---- 2019-03-12 21:59:27 UTC - David Kjerrumgaard: @Sam Leung Which provider are you looking to support? ---- 2019-03-12 23:23:58 UTC - Sanjeev Kulkarni: Anyone know how to enable stream storage for a configuration where bookkeeper is running as a seperate service? Just turning on extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent doesnt seem to work ---- 2019-03-12 23:24:04 UTC - Sanjeev Kulkarni: @jia zhai @Sijie Guo ^^^ ---- 2019-03-13 00:57:55 UTC - jia zhai: Oh, sorry, missed one change. ---- 2019-03-13 03:05:06 UTC - Samuel Sun: Hi, is there anyway to assign a producer name in python client ? ---- 2019-03-13 03:07:05 UTC - jia zhai: @Sanjeev Kulkarni what is the error that meet? ---- 2019-03-13 03:10:32 UTC - Matteo Merli: use `producer_name` when creating the producer: <https://pulsar.apache.org/api/python/#pulsar.Client.create_producer> ---- 2019-03-13 03:10:59 UTC - Sanjeev Kulkarni: There is no state storage service started in bookie ---- 2019-03-13 03:11:19 UTC - Sanjeev Kulkarni: Essentially if I do telnet bookie 4181 it doesn’t connect ---- 2019-03-13 03:14:15 UTC - jia zhai: There should be some logs related to the error in bookie. ---- 2019-03-13 03:18:18 UTC - Sanjeev Kulkarni: I don’t see any errors ---- 2019-03-13 03:18:23 UTC - Sanjeev Kulkarni: The bookie is running fine ---- 2019-03-13 03:18:47 UTC - Sanjeev Kulkarni: The function instance is just stuck in the future in createstatetable function for ever ---- 2019-03-13 03:39:52 UTC - Samuel Sun: thanks, just want to verify is it working well ? I use producer = client.create_producer(‘<persistent://proxy-tenant/proxy-namespace/proxy-v0’,producer_name=‘producer_name_single’>), but I can’t get producer name from <https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/api/PulsarDecoder.java#L88> by cmd.getProducer().getProducerName() ---- 2019-03-13 03:46:31 UTC - Samuel Sun: I am using proxy between producer and broker, and trying to get producer name from proxy side. ---- 2019-03-13 04:23:06 UTC - Samuel Sun: @Matteo Merli ---- 2019-03-13 04:26:44 UTC - bhagesharora: Hello everyone, What is the reason for SocketTimeoutException I am creating one data source using below command but its returning null and SocketTimeoutException bhagesharora93@pulsar-setup-2-3-0-n1-standalone:~/apache-pulsar-2.3.0$ ./bin/pulsar-admin source create --tenant test-tenant --namespace test-namespace --name generator --destinationTopicName generator_test --source-type data-generator null Reason: javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: Read timed out ---- 2019-03-13 05:07:24 UTC - Sijie Guo: any errors in bookie log? ---- 2019-03-13 05:14:11 UTC - Shivji Kumar Jha: @Sijie Guo ANy plans to merge this today? <https://github.com/apache/pulsar/pull/3766> ---- 2019-03-13 05:14:50 UTC - Sijie Guo: @Shivji Kumar Jha I am holding on merging this until <https://github.com/apache/pulsar/pull/3752> is merged. ---- 2019-03-13 05:15:08 UTC - Sijie Guo: #3752 is introducing a SchemaDefinition class. ---- 2019-03-13 05:15:29 UTC - Sijie Guo: so I will add the json definition to SchemaDefinition class. It will make the interfaces much clearer ---- 2019-03-13 05:15:42 UTC - Sijie Guo: #3752 is pending for a review form Matteo ---- 2019-03-13 05:18:45 UTC - Shivji Kumar Jha: @Sijie Guo Okay i will use your fork meanwhile maybe and test my use case. Thank you! Looking forward to the merge :smile: ok_hand : Sijie Guo ---- 2019-03-13 05:24:01 UTC - Sijie Guo: @Shivji Kumar Jha than you for your patience ---- 2019-03-13 05:46:08 UTC - Sanjeev Kulkarni: ---- 2019-03-13 05:46:20 UTC - Sanjeev Kulkarni: ---- 2019-03-13 05:46:28 UTC - Sanjeev Kulkarni: none that I can see ---- 2019-03-13 06:16:13 UTC - Kev Jackson: @Kev Jackson has joined the channel ---- 2019-03-13 06:20:18 UTC - Kev Jackson: good morning - in the past I have signed an Apache ICLA file, and we used a commit-then-review process, scanning the pulsar docs, it looks like a standard github pull request model. Am I good to clone the repo and submit pull requests then (only have a minor change that I noticed while reviewing the code for work)? ---- 2019-03-13 06:25:35 UTC - Matteo Merli: @Kev Jackson You’re good to go, the Apache ICLA is only required for committers. Anyone can open PRs in github ---- 2019-03-13 06:26:49 UTC - Kev Jackson: thanks, I'll open a PR in a moment - given the direction we're heading at work, could be first of many... ---- 2019-03-13 06:28:33 UTC - Matteo Merli: :ok_hand: ---- 2019-03-13 07:18:07 UTC - Kev Jackson: @Matteo Merli I've submitted my PR, I think I've followed the structure of the PR template and answered correctly - happy to be corrected if I've not followed process in some fashion ---- 2019-03-13 08:39:39 UTC - Sébastien de Melo: Yes, no problem up to now. We are only using 1, for routing the messages ----
