2019-07-30 11:28:35 UTC - Jacob O'Farrell: Hi all! In terms of resources to run a Pulsar cluster - are there any recommended minimums with respect to what to set the java xms / xmx parameters to etc? I notice some rather large options are set/suggested in the Kubernetes example yamls, however I was unsure if we can get away with less for a dev cluster or similar? (e.g. <https://github.com/apache/pulsar/blob/master/deployment/kubernetes/aws/broker.yaml#L28>) ---- 2019-07-30 13:21:50 UTC - Richard Sherman: I'm trying to increase redundancy in our 5 node bookkeeper cluster and have increased the ensemble and write quorum sizes from 2 to 3 in each nodes configuration. I've then restarted each bookie. What I am seeing now when I look at a new ledger's meta data is that some have the new values and some are using the old value. What am I missing here? ---- 2019-07-30 13:32:02 UTC - Sijie Guo: you mean old ledgers using old values? ---- 2019-07-30 13:34:12 UTC - Richard Sherman: No new ledgers with the old value ---- 2019-07-30 14:49:13 UTC - Howard Zhang: Thanks for replying @jia zhai! Yes, I do know the ledgerId and entryId. The problem here is that the documentation for admin REST API is wrong. From the doc, `/admin/v2/non-persistent/{tenant}/{namespace}/{topic}/subscription/{subName}/resetcursor`, I'm not sure how should I pass the ledger ID and entry ID to this endpoint. ---- 2019-07-30 15:22:50 UTC - Grant Wu: Created <https://github.com/apache/pulsar/issues/4855> +1 : David Kjerrumgaard ---- 2019-07-30 15:54:51 UTC - Grant Wu: <https://github.com/apache/pulsar/issues/4856> Filed a bug for this :confused: ---- 2019-07-30 17:41:58 UTC - Ambud Sharma: is there a way to limit (quota) bandwidth used autorecovery so ledger replication backfill traffic doesn't impact production ---- 2019-07-30 18:35:03 UTC - Ryan Brereton: @Ryan Brereton has joined the channel ---- 2019-07-30 19:47:29 UTC - Alexandre DUVAL: Hi, there is authentication on pulsar WebSockets API? ---- 2019-07-30 19:50:34 UTC - Chris Bartholomew: Yes, I use token-based authentication on the WebSockets API. ---- 2019-07-30 19:53:57 UTC - Alexandre DUVAL: How to do you pass token to WS? I don't see documentation about on the official documentation. ---- 2019-07-30 19:55:48 UTC - Chris Bartholomew: In the Authorization header ```Bearer <token>``` ---- 2019-07-30 19:58:04 UTC - Alexandre DUVAL: As simple as logic. Thanks. ---- 2019-07-30 20:35:18 UTC - Tarek Shaar: Can someone please provide me with examples (resources) on setting the book keeper configurations of Ensemble Size (E), Write Quorum Size (Qw) and Ack Quorum Size (Qa)? ---- 2019-07-30 22:07:09 UTC - Vikram Dham: @Vikram Dham has joined the channel ---- 2019-07-31 00:09:23 UTC - Devin G. Bost: Is there a way to vary the behavior of different instances of a function? For example, for a backfill operation, if we want to split the bulk operation of moving data from a large table into different parts (like into different ranges of rows from the table) for the different function instances (like how parallelization works in Apache Sqoop), is there a way to do that? ---- 2019-07-31 00:13:31 UTC - Ali Ahmed: you can handle differences with the function context it has an id paramater ---- 2019-07-31 00:14:03 UTC - Ali Ahmed: it can help configure the behavior of functions within an instance range ---- 2019-07-31 03:35:26 UTC - jia zhai: For the POST content, how about passed in Json format. e.g. { ‘ledgerId’: value1, ‘entryId’: value2 } ---- 2019-07-31 07:41:50 UTC - Yi Tang: @Yi Tang has joined the channel ---- 2019-07-31 07:44:53 UTC - Richard Sherman: In your configuration file bk_server.conf ``` #### Replication Settings dlog.bkcEnsembleSize=3 dlog.bkcWriteQuorumSize=3 dlog.bkcAckQuorumSize=2 ``` ---- 2019-07-31 07:55:29 UTC - Yi Tang: Say i have some data *serialized* with AVRO , i want to send them to a topic with corresponding schema, now the client only export *send* method with domain Object parameter which requires me to *deserialize* these bytes data before. Is there a way to send these raw bytes directly? Or we should add a method for this senario. ---- 2019-07-31 07:59:28 UTC - Yi Tang: anyone has a clue to help? thanks ---- 2019-07-31 08:03:12 UTC - Guillaume Braibant: This can help you :
<https://pulsar.apache.org/docs/en/concepts-schema-registry/> You can configure your producer to send raw bytes. You will have to make sure that each message payload (in bytes) corresponds to one 'record' that fit your Avro Schema. Then, on the consumer side, you can configure the consumer with the corresponding AVRO schema and consume domain Object. Pulsar will do the translation by itslef. ---- 2019-07-31 08:03:57 UTC - Guillaume Braibant: PS : This is the first case in the documentation where 'type-safety' is handled on the client side with no schema enforced at topic level. ---- 2019-07-31 08:08:18 UTC - Yi Tang: what i mean is that i want to make use of the topic schema mechanism, for producing and consuming both. the exported send method will encode the object to a bytebuffer named content, now i have the content already, i want to go continue with the same way. ---- 2019-07-31 08:20:24 UTC - Yi Tang: i tried to use non-schema producer which indicate Schema.BYTES , however, it will trim the original AVRO schema in message. ---- 2019-07-31 08:22:45 UTC - Guillaume Braibant: Ok now I understand better your problem. ---- 2019-07-31 08:26:03 UTC - Guillaume Braibant: For this, I am unsure about the answer. I would say that it is OK to use the schema registry to register your AVRO Schema (enforce schema at topic level) then send your AVRO record as raw bytes. Sorry for not being more helpful :slightly_smiling_face: ---- 2019-07-31 08:31:19 UTC - Yi Tang: yeah, i think so. i am confused if there is a native way to do like this ---- 2019-07-31 09:09:10 UTC - Shubham Maheshwari: if my ackTimeout is set to 10 mins and the message I am consuming is taking 30 mins (for e.g.). The broker will resurface the message after 10 mins, even though one of the consumer is still working on it. Is there a way to avoid this scenario? I've created an issue in github as well for this: <https://github.com/apache/pulsar/issues/4861> ----
