[ https://issues.apache.org/jira/browse/ATLAS-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436927#comment-15436927 ]
Manikumar Reddy commented on ATLAS-1131: ---------------------------------------- Kafka Producer logs these warn messages, when it is not able to fetch topic metadata. The reasons for this are unavailability of Topic metadata/Leader/Topic not existing etc. Kafka default settings should be fine for production deployments. Also the warn messages should not be a problem on a healthy cluster. Kafka producer metadata fetch logic is complex and difficult to bring any changes. During integration/unit tests, If we want to reduce the the log volume, then we can decrease the max.block.ms to 30 and increase retry-backoff-ms to 1000. > Atlas hooks produce excess Kafka logs if Kafka topic is not created already > --------------------------------------------------------------------------- > > Key: ATLAS-1131 > URL: https://issues.apache.org/jira/browse/ATLAS-1131 > Project: Atlas > Issue Type: Bug > Reporter: Vimal Sharma > Assignee: Vimal Sharma > Attachments: ATLAS-1131.patch > > > Hooks for Atlas publish messages to a Kafka topic named ATLAS_HOOK. If this > topic is not present and Atlas does not have permission to create this topic, > Kafka produces excessive logs as below (example in /tmp/hive/hive.log). > 2016-08-22 06:43:47,655 WARN [kafka-producer-network-thread | producer-1]: > clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while > fetching metadata with correlation id 1177 : > {ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION} > 2016-08-22 06:43:47,756 WARN [kafka-producer-network-thread | producer-1]: > clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while > fetching metadata with correlation id 1178 : > {ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION} > 2016-08-22 06:43:47,858 WARN [kafka-producer-network-thread | producer-1]: > clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while > fetching metadata with correlation id 1179 : > {ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION} > 2016-08-22 06:43:47,961 WARN [kafka-producer-network-thread | producer-1]: > clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while > fetching metadata with correlation id 1180 : > {ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION} > 2016-08-22 06:43:48,062 WARN [kafka-producer-network-thread | producer-1]: > clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while > fetching metadata with correlation id 1181 : > {ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION} > 2016-08-22 06:43:48,165 WARN [kafka-producer-network-thread | producer-1]: > clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while > fetching metadata with correlation id 1182 : > {ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION} > 2016-08-22 06:43:48,265 WARN [kafka-producer-network-thread | producer-1]: > clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while > fetching metadata with correlation id 1183 : > {ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION} > 2016-08-22 06:43:48,366 WARN [kafka-producer-network-thread | producer-1]: > clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while > fetching metadata with correlation id 1184 : > {ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION} > 2016-08-22 06:43:48,467 WARN [kafka-producer-network-thread | producer-1]: > clients.NetworkClient (NetworkClient.java:handleResponse(600)) - Error while > fetching metadata with correlation id 1185 : > {ATLAS_HOOK=UNKNOWN_TOPIC_OR_PARTITION} -- This message was sent by Atlassian JIRA (v6.3.4#6332)