2018-10-19 09:31:32 UTC - gotham: @gotham has joined the channel ---- 2018-10-19 16:24:32 UTC - Matteo Merli: @Matti-Pekka Laaksonen
1. heap and direct memory are separate memory regions in JVM. The sum needs to be lower than the system memory, with room for OS as well. 2. If you have memory available, you can give a big portion to JVM. This will also help with GC frequency. Otherwise it really depends on number of topics and throughput. For small use cases, a small amount of memory (eg: 256MB) is more than enough. Otherwise, we’d typically use north of 16GB heap and 16 GB direct memory sizing, just to be on safe side. 3. With 16GB memory, I’d do 6GB heap and 6GB direct mem, leaving 4GB for the OS. In any case, we have an active effort to simplify all the memory configurations and have a better out-of-the-box default setting that can work well in almost all conditions. Please take a look at <https://github.com/apache/pulsar/wiki/PIP-24%3A-Simplify-memory-settings> . This is planned for 2.3 release > Oh, and I could swear that I’ve seen a guide to upgrading a running Pulsar cluster, but I can’t seem to find it anymore. Anyone know about this? It was actually never merged :slightly_smiling_face: <https://github.com/apache/pulsar/pull/2048> TL;DR: rolling upgrade of each component with health check after each node upgrade. No downtime. Rollbacks are always guaranteed to be possible as well. ---- 2018-10-19 18:45:41 UTC - Emma Pollum: @Emma Pollum has joined the channel ---- 2018-10-19 18:46:40 UTC - Emma Pollum: Hey team! I was wondering if someone could explain to me how Pulsar chooses which node to assign a topic to. Is it possible to force a node to take a specific topic? ---- 2018-10-19 19:00:36 UTC - Matteo Merli: @Emma Pollum There’s a doc page that explains a bit around the assignment: <http://pulsar.apache.org/docs/en/administration-load-distribution.html> There is no direct way to specify a broker for a specific topic since this assignment is dynamic by nature. The only way is to use broker isolation, to force a group of topics to be on a set of brokers (with fallback policies too) <http://pulsar.apache.org/docs/en/pulsar-admin.html#ns-isolation-policy> ---- 2018-10-19 19:12:38 UTC - Emma Pollum: Awesome, thank you for the information! ---- 2018-10-19 19:47:01 UTC - Ryan Samo: Hey guys, thanks again for all the knowledge. Really helping make headway. One question, on a consumer there is the receiver_queue_size parameter. Is that queue held on or off heap? ---- 2018-10-19 19:47:29 UTC - Matteo Merli: Off heap too ---- 2018-10-19 19:47:58 UTC - Ryan Samo: :+1: ---- 2018-10-19 19:49:23 UTC - Matteo Merli: It’s currently measured in number of messages, you can reduce size to reduce worse case mem usage. Ideally we’ll make that bytes size bound in near term ---- 2018-10-19 19:51:35 UTC - Ryan Samo: I’m actually working with machines that have 512GB RAM so thinking of cranking that number way up actually smile : Matteo Merli, Karthik Ramasamy ---- 2018-10-20 00:43:39 UTC - Ryan Samo: Btw guys I reached 1.1 million messages per sec. today at 100 bytes each! Thanks to you all for your help! ---- 2018-10-20 00:44:17 UTC - Matteo Merli: :tada: tada : Ali Ahmed, Karthik Ramasamy, Sijie Guo ---- 2018-10-20 00:45:02 UTC - Matteo Merli: You should write a blog post! :wink: ---- 2018-10-20 01:20:46 UTC - Rodrigo Malacarne: @Matteo Merli and @Ryan Samo, the blog post idea is really interesting :grin: ---- 2018-10-20 01:21:46 UTC - Rodrigo Malacarne: Especially the cluster setup and config... ---- 2018-10-20 01:22:35 UTC - Rodrigo Malacarne: @Ryan Samo, the topic is non-persistent, right? ---- 2018-10-20 01:48:36 UTC - Ali Ahmed: @Rodrigo Malacarne I think it’s persisted ---- 2018-10-20 02:22:42 UTC - Ryan Samo: Yeah persistent! On heavy hardware but smoking! I might have to put something together. :) ---- 2018-10-20 02:23:03 UTC - Ryan Samo: And not SSD, SAN storage lol ---- 2018-10-20 07:06:39 UTC - kevin: @kevin has joined the channel ----
