2020-05-05 10:40:05 UTC - Matej Šaravanja: Hi :slightly_smiling_face: I'm new here and need you to help me understand something.
Here's the thing. I work at a company where we've built AI platform for running intelligent assistants for various activities (e.g. banking, medicine, telecoms etc.). Platform is a bunch of microservices written in Python/Java/Go which are communicating with eachother through `Kafka` (messages are relatively small, biggest ones for most of users don't exceed 5 kilobytes). This month we've launched our biggest assistant yet with 100k+ users. Now, Kafka did a really good job, it didn't break nor slowed down. But while I was calculating how many brokers we need, I added a few more because of Kafka's scalability issues (it can't be constant uptime due to rebalancing and switching partitions to newly added brokers etc.). Then I started researching Pulsar and really liked it. I ran few tests comparing Kafka and Pulsar latency and throughput. Test was running on my laptop (local deployments of Kafka and Pulsar, 1 broker for each, 1 producer, 1 consumer). Messages were relatively small marking only id of message sent and datetime of the moment message was produced. I used python client as a wrapper for both Kafka and Pulsar to get closer to actual conditions of platform. Pulsar came out as an absolute winner in latency, but Kafka won in throughput. I was producing 3 times 1 million, 3 times 2 million, once 4 million, once 6 million and once 10 million messages. My question to you is, do you maybe know `why this drop happened in Pulsar's throughput between 1 and 2 million messages`? P.S. I'm aware this is not an absolutely proper test, I will run it inside cluster to meet real conditions :slightly_smiling_face: Images are below (Note: `blue line=Kafka`, `red line=Pulsar`) ---- 2020-05-05 10:40:14 UTC - Matej Šaravanja: ---- 2020-05-05 10:58:45 UTC - Alexandre DUVAL: How to create a local release bin from sources repository? ---- 2020-05-05 11:52:59 UTC - Rattanjot Singh: Hi, I am trying to enable tls encryption. Getting this error while executing `bin/pulsar-client produce test --messages "hello-from-beijing-1" -n 1` ```11:49:54.227 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [topic] [null] Error connecting to broker: java.lang.IllegalArgumentException: port out of range:-1``` but `bin/pulsar-admin clusters get us-west` works client.conf configuration webServiceUrl=<https://localhost:8443> brokerServiceUrl=<pulsar+ssl://localhost:6651> tlsEnableHostnameVerification=false tlsTrustCertsFilePath=path_to_file useTls=true ---- 2020-05-05 12:07:29 UTC - Alexandre DUVAL: I mean create a tgz as bin archive downloadable from official website from sources ---- 2020-05-05 12:12:41 UTC - Alexandre DUVAL: got it: distribution/server/target ---- 2020-05-05 12:44:39 UTC - Frank Kelly: I'm going to throw out a wild guess that one of the components e.g. broker or bookie is maxed out for CPU, I/O or the like. Have you made any measurements of any of the components under test to look for a bottleneck? As with all stress tests of any system the setup of your test configuration and the test strategy are critical to understand. I'd say you would need to share both detailed sets of information before you got anything beyond (intelligent) guesses. ---- 2020-05-05 12:48:10 UTC - alex kurtser: Hello I try to figure out which topic has a big backlog with unacknowledged messages and leads to oversized disks on the bookepers. . If i run pulsar-admin topics stats on some topic i get for example this output { "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "averageMsgSize" : 0.0, "storageSize" : 756961890, "backlogSize" : 94277718, "publishers" : [ ], "subscriptions" : { "test" : { "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "msgRateRedeliver" : 0.0, "msgBacklog" : 1826, "blockedSubscriptionOnUnackedMsgs" : false, "msgDelayed" : 0, "unackedMessages" : 0, "msgRateExpired" : 0.004703330824189039, "lastExpireTimestamp" : 1588682066357, "consumers" : [ ], "isReplicated" : false } }, "replication" : { }, "deduplicationStatus" : "Disabled", "bytesInCounter" : 0, "msgInCounter" : 0 } ---- 2020-05-05 12:49:29 UTC - alex kurtser: So my question is what the difference between backlogsize and msgbacklog? ---- 2020-05-05 12:51:25 UTC - alex kurtser: Is here some indicator that can have a glue to the problem that the data is not deleted from the bookeper despite the fact that we have 24 h retention time? ---- 2020-05-05 13:12:43 UTC - Adelina Brask: Hej guys, I am starting 6 Netty HTTP sources on a 3 node cluster. The first 2-3 sources are accepting json messages, but for the last i get the following. It must be something related to the cluster setup ? Is there a parameter limiting how many HTTP netty connections I can run? Starting the sources in localrun works fine. "`13:08:26.962 [nioEventLoopGroup-3-1] ERROR <http://org.apache.pulsar.io|org.apache.pulsar.io>.netty.http.NettyHttpServerHandler - Error when processing incoming data` `java.lang.NoClassDefFoundError: org/apache/pulsar/io/netty/http/NettyHttpServerHandler$NettyHttpRecord` `at <http://org.apache.pulsar.io|org.apache.pulsar.io>.netty.http.NettyHttpServerHandler.channelRead0(NettyHttpServerHandler.java:88) ~[pulsar-io-netty-2.5.0.nar-unpacked/:?]` `at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) ~[netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326) [netty-codec-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300) [netty-codec-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]` `at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.43.Final.jar:4.1.43.Final]` `at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]` `Caused by: java.lang.ClassNotFoundException: <http://org.apache.pulsar.io|org.apache.pulsar.io>.netty.http.NettyHttpServerHandler$NettyHttpRecord` `at <http://java.net|java.net>.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_252]` `at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_252]` `at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_252]` `... 25 more"` ---- 2020-05-05 14:01:33 UTC - Ram: @Ram has joined the channel ---- 2020-05-05 14:02:24 UTC - Matej Šaravanja: @Frank Kelly No, unfortunately I didn't make any measurements. Do you have any suggestion on how to do the Pulsar testing and are there any existing solutions? Only thing I was watching while testing were RAM and CPU which neither hit the roof. ---- 2020-05-05 14:05:10 UTC - Frank Kelly: I would suggest documenting both your Pulsar and Kafka setups and then reshare with this group. I'm new to Pulsar myself but I am an "old hat" at performance/scalability testing across lots of systems and complex distributed systems are hard to debug without being hands-on - with documentation of your setup and test cases more people here could assist. ---- 2020-05-05 14:15:40 UTC - JG: Nobody knows why it takes o luch time to ake the function working on incoming messages ? ---- 2020-05-05 14:40:51 UTC - Matej Šaravanja: Ok, will do with next test :slightly_smiling_face: thanks for suggestions! ---- 2020-05-05 14:52:21 UTC - Shivji Kumar Jha: My guess is msgBacklog would be the number of messages and backlogSize would be the number of bytes! ---- 2020-05-05 14:55:41 UTC - Shivji Kumar Jha: @Matej Šaravanja Maybe there are some cues here? <https://kafkaesque.io/performance-comparison-between-apache-pulsar-and-kafka-latency/> ---- 2020-05-05 15:10:36 UTC - alex kurtser: Hi If i have a namespace with message ttl of -1 day and some topic already has lets say 10000 unacknowledged messages . So if i change the ttl on that namespace to 60 sec . In this case is the new ttl setting applied to the existing messages or only to the new messages that will arrive to the topic after the ttl changing? ---- 2020-05-05 15:21:21 UTC - Robert Yokota: @Robert Yokota has joined the channel ---- 2020-05-05 15:48:01 UTC - Matej Šaravanja: Will look at it, thanks! ---- 2020-05-05 17:30:29 UTC - alex kurtser: Sorry for disturbing , but we continue to investigate our problem with ledgers size and retention. Now we paid attention for something very strange. In some namespaces we have topics with / symbol in its names, for example topic/test , so the full path to that topic is tenant/namespaces/topic/test. Now, these topics are invisible for cli, we can’t get them, cannot list them but if we get their statistics directly by cli against the topic’s full path we can see that that have a large storage size. Currently it seems that other pulsar api don’t see them correctly and probably can’t do the retention on these strange topics ---- 2020-05-05 17:31:31 UTC - alex kurtser: So currently we don’t really know how many and where we have these topics Maybe somebody knows how to find out the all these topics ? Maybe by searching in the zookeper? ---- 2020-05-05 17:32:09 UTC - alex kurtser: And how can we prevent creating topics with illegal symbols in the names ? ---- 2020-05-05 17:34:34 UTC - Sijie Guo: mvn clean install -DskipTests ---- 2020-05-05 17:44:04 UTC - Sijie Guo: msgBacklog is the number of messages that are not consumed by a subscription. Backlogsize is the total bytes of number of messages that are not consumed by all the subscriptions. StorageSize is the total bytes of all the data of a given topic. Messages are deleted ledger by ledger. After ledgers are deleted, bookies have background thread to reclaim the disk space of the deleted ledgers. I have recorded a TGIP episode for talking about the lifecycle of messages. If you are interested in learning more about the details, you can check out this - <https://www.youtube.com/watch?v=R197TYYFaiI&feature=youtu.be> +1 : Shivji Kumar Jha ---- 2020-05-05 17:45:37 UTC - Sijie Guo: Are you running functions using ProcessRuntime? ---- 2020-05-05 17:47:23 UTC - Sijie Guo: ttl is about moving subscription cursors. If you change the ttl on that namespace to 60 sec, if the consumers of that namespace don’t consume the messages. The messages will be marked as TTL-ed. ---- 2020-05-05 17:50:08 UTC - Sijie Guo: You can find all the topics in zookeeper under /managed-ledgers path. ---- 2020-05-05 17:50:37 UTC - Sijie Guo: I thought we added the change to prevent people using “/” in topic name. ---- 2020-05-05 17:50:59 UTC - Sijie Guo: Because “/” will confuse Pulsar whether it is a v1 topic name or a v2 topic name. ---- 2020-05-05 17:55:38 UTC - alex kurtser: Thanks :pray: ---- 2020-05-05 17:56:13 UTC - alex kurtser: Do you know what is the internalevents under the topic? ---- 2020-05-05 17:56:45 UTC - alex kurtser: i see a lot of these even on the deleted tooics ---- 2020-05-05 17:57:46 UTC - alex kurtser: For example look On the first pic you can see that the topic doesn’t exist ---- 2020-05-05 17:58:37 UTC - alex kurtser: ---- 2020-05-05 17:59:16 UTC - alex kurtser: ---- 2020-05-05 18:00:00 UTC - alex kurtser: On the second pic if i am adding /internalevents i can see it and even to see that it takes a lot of sotrage ---- 2020-05-05 18:16:41 UTC - Alex Yaroslavsky: @Sijie Guo in what version should pulsar prevent "/" in topic name? We use 2.5.0 at the moment and as you can see, they are created without an issue ---- 2020-05-05 18:34:49 UTC - Raffaele: public static final Pattern NAMED_ENTITY_PATTERN = Pattern.compile("^[-=:.\\w]*$"); ---- 2020-05-05 18:35:15 UTC - Raffaele: <https://github.com/apache/pulsar/blob/8e1d00ffabf3cf75ec69c504334934a3b4caa4af/pulsar-common/src/main/java/org/apache/pulsar/common/naming/NamedEntity.java> ---- 2020-05-05 18:36:37 UTC - Raffaele: The regex is there, it seems like has always been ---- 2020-05-05 18:41:37 UTC - Kirill Merkushev: Hey, I’ve faced weird classpath issue when running in cluster mode, so I prepared a little project to show an issue Pulsar can’t find my SerDe implementation, even its same package as the function (and it finds the function without issues) <https://gist.github.com/lanwen/769a1204b26f8392c3929c7457a980e2> project is here <https://github.com/lanwen/pulsar-functions-example> and the test class <https://github.com/lanwen/pulsar-functions-example/blob/master/tests/src/test/java/ru/lanwen/pulsar/functions/IntegrationTest.java#L132> starting this test its expected to see both input and output, but only that exception is thrown. Other modes with another 3 methods work fine. Maybe you would have some thoughts? ---- 2020-05-05 18:42:56 UTC - Raffaele: <https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/naming/NamespaceName.java> same here ---- 2020-05-05 20:10:39 UTC - Zarrar: Hi, I am trying to deploy function in pulsar 2.5.0 deployed on kubernetes, the pod keeps crashing with 'Reason: HTTP 401 Unauthorized'. Previously in the older version I had the following varibales set, but they don't seem to work. ```PF_clientAuthenticationPlugin PF_clientAuthenticationParameters``` ---- 2020-05-05 20:32:00 UTC - Yifan: Thanks! ---- 2020-05-05 20:42:30 UTC - Alexandre DUVAL: There is a way to add proxy component to standalone? I'm trying to cover all cases issued from <https://github.com/apache/pulsar/pull/6870> but quite hard to do :/. ---- 2020-05-05 21:02:57 UTC - Chris Hansen: @Chris Hansen has joined the channel ---- 2020-05-05 21:31:32 UTC - Chris Hansen: :wave: Hi all. New user question incoming. It would seem for the Java client that `Schema.JSON(class)` only works for classes with no Jackson annotations (no-arg constructor among other restrictions), or classes using `<http://org.apache.pulsar.shade.com|org.apache.pulsar.shade.com>.fasterxml.jackson.annotation.*` . As a consequence, it’s difficult to reuse Java classes and it effectively makes the shaded classes a part of the public API for the Pulsar Java client (they work if you use them in your POJOs in the 2.5.1 Java client). These are only POJOs after all, so is this a bug or a feature? More broadly, how are users intended to use the JSON schema? <https://pulsar.apache.org/docs/en/client-libraries-java/#schema-example|The docs> seem to imply that our classes should be annotation-free. ---- 2020-05-05 23:29:48 UTC - Alexander Ursu: Hi, do the Presto sql-workers expose Prometheus metrics like how the rest of the system does? Thought it may be undocumented, as the proxy metrics also seem to be hard to find. ---- 2020-05-05 23:41:02 UTC - Sijie Guo: Interesting. In order to clean the storage up, you can use zookeeper to find the list of topics under “test542”. Then use bin/pulsar-admin tool to delete those topics. You can write a bash script to do so. ---- 2020-05-05 23:41:22 UTC - Sijie Guo: I think there were changes in master which will be relased in 2.6.0. @Penghui Li can help confirm that. ---- 2020-05-06 00:18:59 UTC - Kirill Merkushev: So it somehow can’t be loaded by classloader ---- 2020-05-06 00:32:43 UTC - Kirill Merkushev: why it ever using nar classloader instead of URI one? ---- 2020-05-06 01:15:22 UTC - Kirill Merkushev: Ok, actually after debugging for a while I found that it fails actually in `org.apache.pulsar.functions.source.TopicSchema#getSchema(java.lang.String, java.lang.Class<?>, java.lang.String, boolean)` where it’s trying to load my class with a system class loader, which is obviously doesn’t contain my class. What I don’t understand is why it ever goes into that direction ---- 2020-05-06 01:26:25 UTC - Jerry Peng: So I think in 2.5 you also have to explicitly specify the ```authProvider``` For the KubernetesRuntime ---- 2020-05-06 01:26:35 UTC - Jerry Peng: <https://github.com/apache/pulsar/blob/master/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntimeFactory.java#L111> ---- 2020-05-06 01:27:05 UTC - Jerry Peng: I think in 2.4 in was hard coded to always use a specific FunctionAuthProvider ---- 2020-05-06 01:27:59 UTC - Jerry Peng: You can just set ```authProvider = org.apache.pulsar.functions.auth.KubernetesSecretsTokenAuthProvider``` ---- 2020-05-06 01:28:17 UTC - Jerry Peng: under the kubernetes runtime section of the function worker config ---- 2020-05-06 01:28:34 UTC - Zarrar: ok, let me try it ---- 2020-05-06 01:30:50 UTC - Penghui Li: Looks we do not prevent “/” in the local topic name now. Considering we have another topic name format <persistent://tenant/cluster/namespace/local-name> , so we just can prevent “/” in the local name right? Looks still can’t resolve the confusion of v1 and v2 topic name. ---- 2020-05-06 01:31:12 UTC - Sijie Guo: @Jerry Peng ``` if (StringUtils.equals(this.getFunctionRuntimeFactoryClassName(), KubernetesRuntimeFactory.class.getName()) || getKubernetesContainerFactory() != null) { return KubernetesSecretsTokenAuthProvider.class.getName(); } return null;``` ---- 2020-05-06 01:31:32 UTC - Sijie Guo: I think there was a change in 2.5.0 that broke the logic for checking k8s container. ---- 2020-05-06 01:31:43 UTC - Sijie Guo: so it will cause TokenAuthProvider is not set. ---- 2020-05-06 01:31:53 UTC - Sijie Guo: Addison fix the backward compatibility. ---- 2020-05-06 01:32:07 UTC - Sijie Guo: So the token auth provider is still the default setting. ---- 2020-05-06 01:32:19 UTC - Sijie Guo: The fix is in 2.5.1 ---- 2020-05-06 01:32:34 UTC - Sijie Guo: But I agree setting the auth provider explicitly is probably better. ---- 2020-05-06 01:41:33 UTC - Kirill Merkushev: looks like I’m doing something completely wrong with this SerDe definitions, have a feeling that it fails randomly :smile: ---- 2020-05-06 01:43:31 UTC - Kirill Merkushev: Hello there. Can somebody help me to run pulsar function with consumption from EARLIEST position instead of latest (which seems default) - can’t find that in API or cli ---- 2020-05-06 01:45:48 UTC - Zarrar: ok, so I have tried 2.5.1 for broker, but I am getting the same issue, I will try setting the property ---- 2020-05-06 01:53:39 UTC - Zarrar: @Jerry Peng still not luck I set the authProvider ---- 2020-05-06 01:53:43 UTC - Zarrar: ```functionRuntimeFactoryConfigs: authProvider: org.apache.pulsar.functions.auth.KubernetesSecretsTokenAuthProvider``` ---- 2020-05-06 01:54:05 UTC - Zarrar: was it suppose to be under kubernetesContainerFactory ---- 2020-05-06 02:07:16 UTC - Zarrar: Sorry, upgrading the version fixed it. ---- 2020-05-06 02:07:28 UTC - Zarrar: Thank you for your help ---- 2020-05-06 02:12:01 UTC - Kirill Merkushev: created an issue <https://github.com/apache/pulsar/issues/6883> I think better to discuss anything there ---- 2020-05-06 02:45:51 UTC - Sijie Guo: Cool. Glad to know ---- 2020-05-06 02:47:18 UTC - Sijie Guo: Okay then let’s raise a discussion in the dev@ mailing list to discuss removing the support of v1 topic name. Because this is very confusion. ---- 2020-05-06 02:50:42 UTC - Curtis Cook: i think you need to make a reader consumer for that, not sure if it’s supported in functions ---- 2020-05-06 02:51:20 UTC - Curtis Cook: <https://pulsar.apache.org/docs/en/concepts-clients/> ---- 2020-05-06 03:59:31 UTC - Alex Yaroslavsky: That would be great, thanks! We also noticed that / is allowed in the subscription name, is it safe there or should be prevented as well? ---- 2020-05-06 05:50:49 UTC - Penghui Li: There should be problems when store subscription to zookeeper, it’s better to prevent / for the subscription name. ---- 2020-05-06 06:41:26 UTC - Alex Yaroslavsky: Should I open a bug for this in github? +1 : Penghui Li ---- 2020-05-06 06:45:12 UTC - Penghui Li: Thanks, you can create a issue for tracking this task. ---- 2020-05-06 06:51:42 UTC - Alex Yaroslavsky: Opened two issues: <https://github.com/apache/pulsar/issues/6885> <https://github.com/apache/pulsar/issues/6886> +1 : Penghui Li ---- 2020-05-06 06:59:18 UTC - bin: @bin has joined the channel ---- 2020-05-06 07:09:39 UTC - Penghui Li: You can upgrade to 2.5.1 and enable explicit LAC by setting bookkeeperExplicitLacIntervalInMills=10 in the broker.conf. ---- 2020-05-06 07:51:27 UTC - Adelina Brask: @Sijie Guo Yes . Will ThreadRuntime be bedst for production env with many http connections? ---- 2020-05-06 08:03:18 UTC - Gilles Barbier: even using annotations from `<http://org.apache.pulsar.shade.com|org.apache.pulsar.shade.com>.fasterxml.jackson.annotation.*` is not supported (according to my tests) ---- 2020-05-06 08:10:54 UTC - Adelina Brask: Also, I have function_worker started with broker ---- 2020-05-06 08:14:03 UTC - Adelina Brask: I have changed to ThreadRuntime and I get the same error - 1st source is working , other sources created I get the above error , and "Empty reply from server" when posting json with curl. ---- 2020-05-06 08:30:50 UTC - Adelina Brask: Error is: "08:29:14.820 [nioEventLoopGroup-3-1] ERROR <http://org.apache.pulsar.io|org.apache.pulsar.io>.netty.http.NettyHttpServerHandler - Error when processing incoming data java.lang.NoClassDefFoundError: org/apache/pulsar/io/netty/http/NettyHttpServerHandler$NettyHttpRecord" ---- 2020-05-06 08:30:57 UTC - Rattanjot Singh: Hi. when enabling TLS i see this error in broker logs ```ERROR org.apache.pulsar.common.util.SslContextAutoRefreshBuilder - Execption while trying to refresh ssl Context failed to set cipher suite: [NULL-SHA256] <http://javax.net|javax.net>.ssl.SSLException: failed to set cipher suite: [NULL-SHA256] at io.netty.handler.ssl.ReferenceCountedOpenSslContext.<init>(ReferenceCountedOpenSslContext.java:259) ~[io.netty-netty-handler-4.1.43.Final.jar:4.1.43.Final] at io.netty.handler.ssl.OpenSslContext.<init>(OpenSslContext.java:43) ~[io.netty-netty-handler-4.1.43.Final.jar:4.1.43.Final] at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:346) ~[io.netty-netty-handler-4.1.43.Final.jar:4.1.43.Final] at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:334) ~[io.netty-netty-handler-4.1.43.Final.jar:4.1.43.Final] at io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext.java:471) ~[io.netty-netty-handler-4.1.43.Final.jar:4.1.43.Final] at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:457) ~[io.netty-netty-handler-4.1.43.Final.jar:4.1.43.Final] at org.apache.pulsar.common.util.SecurityUtility.createNettySslContextForServer(SecurityUtility.java:115) ~[org.apache.pulsar-pulsar-common-2.5.0.jar:2.5.0] at org.apache.pulsar.common.util.NettySslContextBuilder.update(NettySslContextBuilder.java:44) ~[org.apache.pulsar-pulsar-common-2.5.0.jar:2.5.0] at org.apache.pulsar.common.util.NettySslContextBuilder.update(NettySslContextBuilder.java:31) ~[org.apache.pulsar-pulsar-common-2.5.0.jar:2.5.0] at org.apache.pulsar.common.util.SslContextAutoRefreshBuilder.get(SslContextAutoRefreshBuilder.java:90) [org.apache.pulsar-pulsar-common-2.5.0.jar:2.5.0] at org.apache.pulsar.broker.service.PulsarChannelInitializer.initChannel(PulsarChannelInitializer.java:67) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0] at org.apache.pulsar.broker.service.PulsarChannelInitializer.initChannel(PulsarChannelInitializer.java:32) [org.apache.pulsar-pulsar-broker-2.5.0.jar:2.5.0] at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final] at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:964) [io.netty-netty-transport-4.1.43.Final.jar:4.1.43.Final]``` ----
