Hello,
I would like to implement possibility to use external kafka server for log
aggregation.
Now twill uses EmbededKafkaServer for that. I think implementation would
look like this:
1. Add ZK path where kafka zk connection string will be stored. There
should should be only one such path per ApplicationMaster
2. Use this path in ApplicationKafkaService while creating
EmbededKafkaService, if there on brokers right now
3. For log aggregation there should be additional nodes in ZK for each
instance with kafka topic's name in it. Something like
"test-app-{UUID}-log". So publisher will send to this topic and consumer
will consume log messages on the job submission machine.
What would you say? Is this sounds ok?
Sergey
