Ranger devs & users – rangertagsync appears be processing from Kafka VERY
slowly.
For example, as seen in the commands below:
* In the 60 seconds between the 1st 2 commands:
* The RANGER_ENTITIES topic grew by 5059
* The offset (from rangertagsync) only moved by 48!
* The lag is getting bigger and bigger.
```
# date; ./bin/kafka-consumer-groups.sh --zookeeper ${zk} --describe --group
ranger_entities_consumer
Wed Nov 8 11:35:23 CST 2017
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
ranger_entities_consumer ATLAS_ENTITIES 0
543278559 585346756 42068197
ranger_entities_consumer_host.domain.com-1510161276586-4537fb87-0
# date; ./bin/kafka-consumer-groups.sh --zookeeper ${zk} --describe --group
ranger_entities_consumer
Wed Nov 8 11:36:25 CST 2017
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
ranger_entities_consumer ATLAS_ENTITIES 0
543278607 585351815 42073208
ranger_entities_consumer_host.domain.com-1510161276586-4537fb87-0
# date; ./bin/kafka-consumer-groups.sh --zookeeper ${zk} --describe --group
ranger_entities_consumer
Wed Nov 8 11:37:27 CST 2017
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
ranger_entities_consumer ATLAS_ENTITIES 0
543278653 585356473 42077820
ranger_entities_consumer_host.domain.com-1510161276586-4537fb87-0
# date; ./bin/kafka-consumer-groups.sh --zookeeper ${zk} --describe --group
ranger_entities_consumer
Wed Nov 8 11:38:30 CST 2017
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
ranger_entities_consumer ATLAS_ENTITIES 0
543278700 585360642 42081942
ranger_entities_consumer_host.domain.com-1510161276586-4537fb87-0
```
--
Sean Roberts
@seano
