GitHub user bsandy999 edited a discussion: 15k TPS with Payload size of 52kb?
Hello Folks,
It's been a month now and I am happy to share my POC experience
with the Pulsar Community. Also have some scaling, stability, and
configs-related concerns to resolve.
With **Standalone** Setup, I got familiar with the basic concepts/terms and
component-wise flow like broker, bookKeeper, and ZooKeeper, etc.
With **Bare Metal** setup, I learned to run Apache Pulsar Cluster on multiple
Nodes. Which results in an increase in TPS too.
My Major concern is to achieve maximum TPS with Stable infra, which is missing
in my current setup.
So basically I am doing a load test with a **52kb** payload size with
**100000** (1 Lakh (i.e 0.1 Million)) Records in a single request. To process
all 1 lakh batches it is taking around **46sec**.
So in short it gives me 4.6k TPS to process 0.1 Million records with 52kb
payload size each. Where I wanted to achieve min TPS of **_15k with a Payload
size of 52kb._**
**_Cluster Stability Concern - Benchmark/Load Test Observations (On running
load test multiple times)_**
- Brokers get down randomly (sometimes OOM issue, sometimes another random
issue like System load, etc)
- Bookie gets down randomly (sometimes OOM issue, sometimes another random
issue like System load, etc)
- The data folder does not get a decrease in size even if messages got consumed
(persistent storage)
- Broker does not get graceful shutdown on stopping it using CLI (to scale down
the number of brokers and to reduce the load on a machine)
All above-mentioned issues are recurring and hence it is impacting POC results.
**_Questions._**
_Kindly suggest the best architecture setup so that setup will be scalable and
easy to troubleshoot_
- A basic checklist to maintain while setting up the Pulsar Cluster
- Basic Server Configuration to achieve mentioned TPS
- Cluster tunning & troubleshooting tips to avoid maintenance time
_Sharing the current setup design on which I am running load the test ( All
machines are in GCP)_
Stack Forum - [Stack Overflow
](https://stackoverflow.com/questions/75518717/apache-pulsar-15k-tps-with-payload-size-of-52kb)
Please go through the above-detailed information and suggest to me the best
solution to resolve my requirement
Thanks in advance!
GitHub link: https://github.com/apache/pulsar/discussions/19578
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
