----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73409/#review223121 -----------------------------------------------------------
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java Line 137 (original), 138 (patched) <https://reviews.apache.org/r/73409/#comment312247> Would this result in an additional call to Kafka broker for each message, to retrieve endOffset? If yes, consider tracking of endOffset to metrics module where the offset will be retrieved only on demand. Rest of the chages look good! This will be a very useful addition. - Madhan Neethiraj On June 7, 2021, 11:12 p.m., Sarath Subramanian wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73409/ > ----------------------------------------------------------- > > (Updated June 7, 2021, 11:12 p.m.) > > > Review request for atlas, Ashutosh Mestry, Deep Singh, Jayendra Parab, Madhan > Neethiraj, Nikhil Bonte, Nixon Rodrigues, Radhika Kundam, and Sidharth Mishra. > > > Bugs: ATLAS-4330 > https://issues.apache.org/jira/browse/ATLAS-4330 > > > Repository: atlas > > > Description > ------- > > In metrics page, in addition to other Notification details. Lag information > of kafka consumer topics will be useful to check the current backed up > messages in Atlas: > > The solution involves using the kafka admin client APIs to get log end offset > of a topic for the consumer ('atlas') > > Config property: > > atlas.notification.consumer.topic.report.frequency.seconds=60 (default value > 3600secs or 1 hour) > > > Diffs > ----- > > common/src/main/java/org/apache/atlas/utils/KafkaUtils.java 167442259 > > dashboardv2/public/js/templates/site/Statistics_Topic_Offset_table_tmpl.html > b4c8d8677 > dashboardv2/public/js/views/site/Statistics.js 32a66691c > > dashboardv3/public/js/templates/site/Statistics_Topic_Offset_table_tmpl.html > b4c8d8677 > dashboardv3/public/js/utils/Enums.js ee80313c2 > dashboardv3/public/js/views/site/Statistics.js 9eb714d45 > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java > 96dc5856a > notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java > af3727df4 > > notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java > 1fb9f9989 > > notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java > 1b486e528 > repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java > beb90e67b > repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java > bea8eb8b5 > > webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java > 5643af95c > > > Diff: https://reviews.apache.org/r/73409/diff/1/ > > > Testing > ------- > > Precommit: > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/624/console > > Manually verified lag information is updated correctly on metrics page and > periodically printed on log > > > Thanks, > > Sarath Subramanian > >
