Thanks a lot Frank!
On Thu, Mar 10, 2022 at 6:08 AM frank <[email protected]> wrote: > Hey Johnny, > Sorry for the delay on replying. You might find this talk I did of use > https://youtu.be/vE7aKZT9r7k > > TL;DR you will need to scale up ZooKeeper (CPU, Memory, Disk) and Brokers > (Memory for sure). > There's probably more I've forgotten that you'll find in the talk. > > In the end we would be creating 500k+ topics per day and since tiered > storage would still require ZooKeeper to keep track of these topics so we > created our own "offloader" to read topics once they are finished and > writes the data to S3. > > -Frank > > > > > ---- On Thu, 24 Feb 2022 05:08:33 -0500 *Johnny Miller > <[email protected] <[email protected]>>* wrote ---- > > Hi - I have a project I am at the design phase in, and it would be > interesting to leverage the million topic scale in Pulsar. I have > experienced problems in similar tech where these limits are > more"hypothetical" rather than real. > > Is this realistic in prod/operationally? Not necessarily a lot of data, > more just lots of topics. > > Any real stories, experiences or war wounds around this level of scale > would be appreciated. > > Thanks, > > Johnny > > > -- > > The information contained in this electronic message and any attachments > to this message are intended for the exclusive use of the addressee(s) and > may contain proprietary, confidential or privileged information. If you are > not the intended recipient, you should not disseminate, distribute or copy > this e-mail. Please notify the sender immediately and destroy all copies of > this message and any attachments. WARNING: Computer viruses can be > transmitted via email. The recipient should check this email and any > attachments for the presence of viruses. The company accepts no liability > for any damage caused by any virus transmitted by this email. > www.digitalis.io > > > >
