suddendust opened a new pull request, #12522:
URL: https://github.com/apache/pinot/pull/12522
This PR adds two metrics in `RealtimeSegmentDataManager`:
1. `REALTIME_LAST_FETCHED_BATCH_SIZE`: This is a new gauge that can assume
the following values:
- 0: When there were no messages in the last batch. This can happen when
there are no messages or very low vol of messages in the stream partition.
- 0: When messages were fetched in the last batch.
- -1: When there were exceptions fetching the message batch from the stream
partition.
This metric can be used to identify the 1st case of no messages as:
`sum by (table) (pinot_server_realtimeLastFetchedBatchSizeCount{}) == 0`. We
can use this source alert to inhibit other alerts. For example, an alert on
realtime ingestion stopped.
**Why can't we reuse `pinot_server_realtimeRowsConsumed`?**
`pinot_server_realtimeRowsConsumed` tracks the number of rows that were
successfully indexed. If there were problems transforming/indexing the row,
those rows aren't counted in this. So it makes it hard to calculate the total
number of rows being fetched from the partition stream.
3. `STREAM_CONSUMER_CREATE_EXCEPTIONS`: If we face exceptions trying to
create a stream consumer.
Testing:
`STREAM_CONSUMER_CREATE_EXCEPTIONS`:
<img width="1225" alt="Screenshot 2024-02-29 at 12 30 43 PM"
src="https://github.com/apache/pinot/assets/84911643/f9427cfc-85ac-43c4-a40c-b33bd7a88198";>
`REALTIME_ROWS_FETCHED`:
<img width="1639" alt="Screenshot 2024-02-29 at 1 34 11 PM"
src="https://github.com/apache/pinot/assets/84911643/5fb3020e-f5df-4272-87cc-fb85d5842e4a";>
<img width="1510" alt="Screenshot 2024-02-29 at 12 35 19 PM"
src="https://github.com/apache/pinot/assets/84911643/fa636fcf-e665-479f-8344-09237f20054c";>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
