[
https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sarath Subramanian updated ATLAS-4330:
--------------------------------------
Description:
In metrics page, in addition to other Notification details. Lag information of
kafka consumer topics will be useful to check the current backup up messages in
Atlas:
!image-2021-06-07-15-03-28-166.png!
It will also be useful to print the lag information periodically in atlas
application.log on a frequency that can be made configurable (print lag every 1
hour, 5min or so).
*atlas.notification.consumer.topic.report.frequency.seconds=60*
{code:java}
2021-06-07 21:56:06,121 INFO - NotificationProcessingStats:
topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0
2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0,
currentOffset=1118, endOffset=1118, lag=0
{code}
was:
In metrics page, in addition to other Notification details. Lag information of
kafka consumer topics will be useful to check the current backup up messages in
Atlas:
!image-2021-06-07-15-03-28-166.png!
It will also be useful to print the lag information periodically in atlas
application.log on a frequency that can be made configurable (print lag every 1
hour, 5min or so).
*atlas.notification.consumer.topic.report.frequency.seconds=60*
{code:java}
2021-06-07 21:56:06,121 INFO - NotificationProcessingStats:
topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0
2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0,
currentOffset=1118, endOffset=1118, lag=0
{code}
> Add Kafka topics lag information on metrics and log
> ---------------------------------------------------
>
> Key: ATLAS-4330
> URL: https://issues.apache.org/jira/browse/ATLAS-4330
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-core
> Affects Versions: 2.1.0
> Reporter: Sarath Subramanian
> Assignee: Sarath Subramanian
> Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: image-2021-06-07-15-03-28-166.png
>
>
> In metrics page, in addition to other Notification details. Lag information
> of kafka consumer topics will be useful to check the current backup up
> messages in Atlas:
> !image-2021-06-07-15-03-28-166.png!
> It will also be useful to print the lag information periodically in atlas
> application.log on a frequency that can be made configurable (print lag every
> 1 hour, 5min or so).
> *atlas.notification.consumer.topic.report.frequency.seconds=60*
> {code:java}
> 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats:
> topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0
> 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats:
> topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
