jugomezv opened a new issue, #10097:
URL: https://github.com/apache/pinot/issues/10097
PR 9994 introduces a per partition metric that measures the time between
when an event was introduced in to the last-hop stream abstraction and the time
Pinot consumes it. In many cases the ingestion pipeline is composed of several
stages and it would be beneficial for the user to know how long the lag is for
an event to cross the full ingestion pipeline and be consumed by pinot. To this
goal we propose to add a new time stamp to generic stream event that would be
creation time that an then be used to emit a per partition metric indicating
end to end ingestion delay (from creation of the event to consumption by Pinot)
Current metric:
(Event)->{(TimeStamp1)->Last hop stream->} ->{(TimeStamp2) Pinot
consumption} Metric = TimeStamp2-TimeStamp1
Proposed Metric:
(Event-TimeStamp1)->{Stream 1}->{Stream 2}->{Ingestion processing
Job}...->{LastHop Stream}->{(TimeStamp2) Pinot consumption} Metric(End to end
Ingestion Delay} = TimeStamp2-TimeStamp1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
