2020-05-09 15:33:30 UTC - Rui Figueiredo: @Rui Figueiredo has joined the channel ---- 2020-05-09 16:32:02 UTC - JG: Hi there, apperently it takes 1 minute to get a pulsar function active after it started: 12:16:02.338 [pulsar-client-io-1-1] INFO Produced message with ID 15:0:-1:0 --> 12:16:*02* And finally after 1 minute: ( 53 secs ) 12:16:55.369 [pulsar-external-listener-3-1] INFO Received message with an ID of 12:0:-1:0 and a payload --> 12:16:*55*
Someone can explain this behavior ? Why it takes so much time to have a pulsar function effective ? ( it started and running correctly wihtout exception ) The broker needs to refresh something ? ---- 2020-05-09 17:14:00 UTC - David Kjerrumgaard: You need to bundle the SerDe class inside the NAR file you create for the function ---- 2020-05-09 19:23:56 UTC - Marcio Martins: @Marcio Martins has joined the channel ---- 2020-05-09 19:27:29 UTC - Marcio Martins: Hey guys, I am trying to setup a small prototype with Pulsar + Flink. I got Flink and Pulsar running in my local k8s, both tested and working, but having troubles with the code, like what to import, etc. Is there any working example repo I could checkout for a quick start? ---- 2020-05-09 21:41:44 UTC - Deepak Sah: Hi everyone, I am using pulsar with go client library. I have 1 ec2 instance for producer(8 cores) and 1 for pulsar(8 cores). I am only able to publish around 60k messages/ second (7kb payload) with `SendAsync` method. Also, tried with multiple goroutines as producers but still couldn't go past 60k. Is it possible to do better than this? ---- 2020-05-09 21:43:47 UTC - Matteo Merli: That's 400 Mbytes/s ---- 2020-05-09 21:44:08 UTC - Matteo Merli: That's probably hitting the disk limit ---- 2020-05-09 22:02:50 UTC - Deepak Sah: Thanks for the quick reply. I decreased the payload size and I can see some improvement. But, when I increase the number of goroutines, number of messages published go down. Also, when I run benchmark for longer duration, the throughput is only around 15k messages/sec ---- 2020-05-10 04:47:28 UTC - Liam Clarke: Assuming you're running Pulsar in AWS also, have you checked Cloudwatch? You may be depleting your burst bucket. Or you may be hitting an IOPs limit. Also check that you're not hitting any network bandwidth limits on the cluster components, or producer and consumer boxes. <https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-io-characteristics.html> ---- 2020-05-10 07:53:00 UTC - Deepak Sah: Thanks @Liam Clarke for the suggestions, it's very helpful. I will check network and disk usage for any bottlenecks. ---- 2020-05-10 08:05:44 UTC - Franck Schmidlin: Thanks, that helps ----
