2019-03-21 09:11:51 UTC - Sébastien de Melo: Hi @David Kjerrumgaard, did you get it to work? ---- 2019-03-21 09:19:26 UTC - Sébastien de Melo: Hi @Sijie Guo, the problem does not occur anymore. It was probably due to an OOM error. I reduced the resources for our pods but forgot to adjust PULSAR_MEM accordingly. My bad ---- 2019-03-21 09:20:40 UTC - Sijie Guo: Glad to hear about that :slightly_smiling_face: ---- 2019-03-21 10:42:56 UTC - Ganga Lakshmanasamy: Hi, We are trying to send collection objects as part of message using AVRO object and it failed due to java reflection problem. Is there any other way the data can be passed? While using JSON schema below error is being thrown Exception in thread "Thread-111" java.lang.RuntimeException: org.apache.pulsar.client.api.SchemaSerializationException: org.apache.pulsar.shade.com.fasterxml.jackson.core.JsonParseException: Illegal character ((CTRL-CHAR, code 2)): only regular white space (\r, \n, \t) is allowed between tokens ---- 2019-03-21 10:43:15 UTC - Ganga Lakshmanasamy: @Sanjeev Kulkarni Can you help me fix this problem ---- 2019-03-21 13:14:03 UTC - Marc Le Labourier: On one of my topic, I get the following error while creating a producer. I understand that I am breaking a quota but any idea on how to solve this type of issue ? ---- 2019-03-21 13:19:08 UTC - Chris Bartholomew: I would recommend Kubernetes. There are lots of parts to a Pulsar deployment and Kubernetes really helps in the management, especially if you want to dynamically scale. If you aren't familiar with Kubernetes, there is definitely a learning curve, but it is worth learning in the long run. ---- 2019-03-21 13:30:38 UTC - Chris Bartholomew: You need to remove some stored messages from the topics in your namespace, since the quota is set at the namespace level. I often find that I have old subscriptions that are storing messages that are not needed, so I would suggest looking at the subscriptions on your topics in the namespace. You can see this using the Pulsar dashboard. Once you have found the subscriptions holding the messages, you can either delete the subscription using ```pulsar-admin topics unsubscribe <topic> -s <subscription>``` or if you want to keep the subscription, but just delete the messages, you can "skip" them using ```pulsar-admin topics clear-backlog <topic> -s <subscription``` ok_hand : Laurent Chriqui, Sébastien de Melo ---- 2019-03-21 13:49:18 UTC - Laurent Chriqui: @Jerry Peng Hello. I tested the timeoutMS option. I couldn’t see it work. I made a function that raises an error 1 second after the timeout 11s vs (10000 Ms) and the message are all processed but then they are not redelivered and reprocessed... Any idea what’s wrong ? ---- 2019-03-21 14:14:44 UTC - David Kjerrumgaard: @Sébastien de Melo I got side tracked getting ready for our demo at Strata next week and haven't had a chance to work on this yet. I will do it this morning ---- 2019-03-21 14:15:19 UTC - Sébastien de Melo: Sure, no problem :wink: ---- 2019-03-21 14:26:12 UTC - Guy Feldman: thanks for your reply. I'm running in kubernetes in our dev environment, was wondering if anyone used kubernetes in prod ---- 2019-03-21 14:40:47 UTC - Jerry Peng: @Laurent Chriqui can you share the command line of how you are creating the function? ---- 2019-03-21 14:44:23 UTC - Laurent Chriqui: @Jerry Peng there it is I didn’t reproduce the topics pattern and the user-config :slightly_smiling_face: ---- 2019-03-21 15:47:20 UTC - Jerry Peng: @Laurent Chriqui weird should work. Let me dig deeper and get back to you ---- 2019-03-21 15:47:42 UTC - Jerry Peng: @Laurent Chriqui what version of Pulsar are you using? ---- 2019-03-21 15:47:53 UTC - Laurent Chriqui: 2.3.0 ---- 2019-03-21 15:47:58 UTC - Laurent Chriqui: thanks ---- 2019-03-21 15:54:05 UTC - Grant Wu: Hey, does anyone here have experience running Pulsar in production, that uses the WebSocket interface? ---- 2019-03-21 15:54:18 UTC - Grant Wu: I’m getting idle timeouts for consumers created that use the WebSocket interface ---- 2019-03-21 15:54:33 UTC - Grant Wu: It appears to be an issue endemic to Jetty: <https://stackoverflow.com/questions/9056159/websocket-closing-connection-automatically> ---- 2019-03-21 15:56:01 UTC - Grant Wu: I think these are some relevant log messages: ---- 2019-03-21 15:56:03 UTC - Grant Wu: ``` 00:06:40.095 [pulsar-web-30-7] INFO org.apache.pulsar.websocket.AbstractWebSocketHandler - [/10.244.2.46:50288] New WebSocket session on topic <persistent://public/grantwu/datasetUpdated> 00:11:46.902 [Scheduler-1490434838] INFO org.apache.pulsar.websocket.AbstractWebSocketHandler - [/10.244.2.46:50288] WebSocket error on topic <persistent://public/grantwu/datasetUpdated> : Timeout on Read 00:11:46.921 [pulsar-client-io-53-7] INFO org.apache.pulsar.websocket.ConsumerHandler - [<persistent://public/grantwu/datasetUpdated/9f5b8773-b2c6-4fc8-8da6-4d29a4239bd6>] Consumer was closed while receiving msg from broker 00:11:46.922 [Scheduler-1490434838] INFO org.apache.pulsar.websocket.AbstractWebSocketHandler - [/10.244.2.46:50288] Closed WebSocket session on topic <persistent://public/grantwu/datasetUpdated>. status: 1001 - reason: Idle Timeout 00:11:46.922 [Scheduler-1490434838] WARN org.apache.pulsar.websocket.ConsumerHandler - [<persistent://public/grantwu/datasetUpdated>] Failed to remove consumer handler ``` ---- 2019-03-21 16:54:45 UTC - vinay Parekar: Thanks guys. we are able to get source working with threadContainerFactory now. ---- 2019-03-21 17:34:47 UTC - Sanjeev Kulkarni: where is this error/ ---- 2019-03-21 17:55:20 UTC - Brian Babcock: @Brian Babcock has joined the channel ---- 2019-03-21 18:18:10 UTC - Jerry Peng: @Laurent Chriqui I am unable to reproduce the behavior you mentioned. When I set ```timeoutMS``` and I raise an exception in my function, the message that cause the exception does get redelivered ---- 2019-03-21 18:18:59 UTC - Jerry Peng: @Laurent Chriqui can you also check what version of the python module pulsar-client you have installed? and what version of python are you running? Python2 or python3? ---- 2019-03-21 18:20:10 UTC - Jerry Peng: @Laurent Chriqui FYI the redelivery is not going to be exactly what you set for timeoutMs. The redelivery is going to be between 1X-2X the value set ---- 2019-03-21 18:20:26 UTC - Laurent Chriqui: I run python3 and pulsar-client-2-3-0post1 ---- 2019-03-21 18:23:00 UTC - Jerry Peng: @Laurent Chriqui are there any errors in function log file? ---- 2019-03-21 18:27:13 UTC - Laurent Chriqui: Only the one I'm raising ---- 2019-03-21 18:28:40 UTC - Laurent Chriqui: When you mean redelivered, you mean that after all the messages in the topic have been processed, the ones raising an exception will come back and be reprocessed by the function? ---- 2019-03-21 18:49:08 UTC - Jerry Peng: @Laurent Chriqui The messages that caused the exception should be redelivered and reprocessed. I think i am able to reproduce what you are seeing. I am wondering if this is an issue with regex topics. Let me investigate further ---- 2019-03-21 19:00:08 UTC - Laurent Chriqui: Yes I thought it might be because of it... ---- 2019-03-21 19:12:19 UTC - Jerry Peng: @Laurent Chriqui ya I was able to reproduce this issue with just the python consumer consuming from a regex. So the problem is in the python consumer. ---- 2019-03-21 19:17:15 UTC - Laurent Chriqui: Good I'm glad you've been able to reproduce. :grin: ---- 2019-03-21 20:21:33 UTC - Brian Babcock: Hi Pulsar folks! I have some questions about best practices, and available configuration knobs, for memory management with the Pulsar Java client for consumers.
The problem I'm encountering is consistently hitting OutOfMemoryError on the client, usually during org.apache.pulsar.client.impl.ConsumerImpl.messageReceived as invoked from netty. I don't have many topics, but have a large message size. I see that I can adjust receiverQueueSize using ConsumerBuilder, but I'm wondering if there are any other settings (perhaps netty configuration via environment variables?) that can be tuned to reduce memory usage. In particular, I'm interested in settings that can control global memory allocation behavior (rather than at a per-topic level), if they exist. Also, it seems that the PIP-24 proposal would provide a lot of what I'm looking for. Is there any estimated timeline for its availability? <https://github.com/apache/pulsar/wiki/PIP-24%3A-Simplify-memory-settings> Thanks in advance for any pointers you can provide! ---- 2019-03-21 20:30:50 UTC - Jerry Peng: @Laurent Chriqui just put up a fix for the problem: <https://github.com/apache/pulsar/pull/3879> ---- 2019-03-21 20:45:31 UTC - Laurent Chriqui: Ok I'll check it out tomorrow! Thank you! ---- 2019-03-21 20:49:16 UTC - Jerry Peng: @Laurent Chriqui let me know if you need instructions on how to build a python client after you pull the fix in ---- 2019-03-21 20:50:03 UTC - Laurent Chriqui: I've compiled it once before. Should be OK. Thank you. ---- 2019-03-21 20:50:37 UTC - Grant Wu: cc @Jonathan Budd this is something we should be cognizant of - we are using regex consumers in our PF as well +1 : Jonathan Budd ---- 2019-03-21 21:48:02 UTC - Matteo Merli: @Brian Babcock Yes, currently the best option is to reduce the receiverQueue size. We have a (very) short term goal to add another improvement that will automatically reduce the prefetching based on throughput need and messages size. That will fix most of the instances of the OOM problems. After that the idea is to have a per-client-instance memory setting that will set the hard-max and will simplify the application configuration. Unfortunately there’s no way in netty itself to put a hard limit on the allocator size. ---- 2019-03-21 22:42:02 UTC - Brian Babcock: Thanks, @Matteo Merli! ---- 2019-03-22 01:17:05 UTC - Zarrar: @Zarrar has joined the channel ---- 2019-03-22 01:57:20 UTC - bossbaby: How can pulsars guarantee the correct order of messages in all cases? ---- 2019-03-22 07:02:21 UTC - bossbaby: @Matteo Merli can you help me? ---- 2019-03-22 08:27:04 UTC - Shivji Kumar Jha: @bossbaby The message order in a topic (or a partition under partitioned topic) is guaranteed. We cant have ordering across topics or across partitions across topics. Does this answer your question? ---- 2019-03-22 08:29:17 UTC - Shivji Kumar Jha: @Brian Babcock How big is your message size? I am afraid we could have bug messages too.. I tested with 2.5 K and assumed it would work for bigger messages! ---- 2019-03-22 08:30:56 UTC - bossbaby: But how does pulsar do that? ---- 2019-03-22 09:02:22 UTC - Shivji Kumar Jha: Hi @Penghui Li, I am wondering if MultiVersionGenericSchemaProvider should be a part of Consumer instance. It should be instantiated in Consumer constructor and then populated in message receive if there was a cache miss. It should then be used for every subsequent consumer.receive(). Do you agree? Instantiating MultiVersionGenericSchemaProvider for every message must be costly i would think. ----
