Re: [I] I/O imbalance across bookies [pulsar]

Wed, 26 Feb 2025 01:12:13 -0800 


Jayer23 commented on issue #24010:
URL: https://github.com/apache/pulsar/issues/24010#issuecomment-2684364234

   > > There are only 3 partitions in the cluster with write rates exceeding 
2MB/s
   > > The producer has Lz4 compression . Does the Prometheus query threshold 
need to be adjusted?
   > 
   > You need to figure out what ledgers contribute most write load on the 
bookies with high write throughput.
   > 
   > How much difference between following querys?
   > 
   > ```
   > sum(pulsar_throughput_in{cluster="YOUR_CLUSTER_NAME"}) / 1024^2
   > 
   > sum(rate(bookie_WRITE_BYTES{group="YOUR_CLUSTER_NAME"}[1m])) / 1024^2
   > ```
   > 
   > Btw, what's query behind this graph?
   > 
   > <img alt="Image" width="846" 
src="https://private-user-images.githubusercontent.com/5058708/417057423-9e4a203d-b14c-4a0d-ad5b-00a9e3bb968f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDA1NjA0NTYsIm5iZiI6MTc0MDU2MDE1NiwicGF0aCI6Ii81MDU4NzA4LzQxNzA1NzQyMy05ZTRhMjAzZC1iMTRjLTRhMGQtYWQ1Yi0wMGE5ZTNiYjk2OGYucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI1MDIyNiUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNTAyMjZUMDg1NTU2WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9OTc1NDMxZWM5MzQzNWQ3ZGJiOTg4NjI1ZWJkN2Q3OTBhNmRiNzg5YWJkYTQ5M2NlZDRmNjQ2ZDkwMTJmMTYzOSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.oZF8prtnzpMAEdaic1epvo1OpfCuy_-1XVlCXp81AIQ";>
   1. sum(pulsar_throughput_in{cluster="YOUR_CLUSTER_NAME"}) / 1024^2
   
   
![Image](https://github.com/user-attachments/assets/e4979922-489f-436b-9a06-1292c73d6578)
   2. sum(rate(bookie_WRITE_BYTES{group="YOUR_CLUSTER_NAME"}[1m])) / 1024^2
   
   
![Image](https://github.com/user-attachments/assets/286dc621-f19f-4cff-b2e7-f39c1eb325db)
   3. We collected host-level disk io metrics，include journal disks and ledger 
disks，found that io-writes of some hosts is much larger than that of others
   <img alt="Image" width="846" 
src="https://private-user-images.githubusercontent.com/5058708/417057423-9e4a203d-b14c-4a0d-ad5b-00a9e3bb968f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDA1NjA0NTYsIm5iZiI6MTc0MDU2MDE1NiwicGF0aCI6Ii81MDU4NzA4LzQxNzA1NzQyMy05ZTRhMjAzZC1iMTRjLTRhMGQtYWQ1Yi0wMGE5ZTNiYjk2OGYucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI1MDIyNiUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNTAyMjZUMDg1NTU2WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9OTc1NDMxZWM5MzQzNWQ3ZGJiOTg4NjI1ZWJkN2Q3OTBhNmRiNzg5YWJkYTQ5M2NlZDRmNjQ2ZDkwMTJmMTYzOSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.oZF8prtnzpMAEdaic1epvo1OpfCuy_-1XVlCXp81AIQ";>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to