My email for apache JIRA is: firstrole...@yandex.ru Sergey
вт, 4 июл. 2017 г. в 8:33, Terence Yim <cht...@gmail.com>: > Sure, you are welcome to do so. You can also update the JIRA with your > proposed change before sending the PR. > > What is your Apache JIRA email? I have to add you to the twill project > before I can assign the JIRA to you. > > Terence > > Sent from my iPhone > > > On Jul 3, 2017, at 12:57 AM, Сергей Филиппов <role...@gmail.com> wrote: > > > > Hi, Terence, > > Thank you for description! Should I assign this issue to me? > > > > > > Sergey > > > > сб, 1 июл. 2017 г. в 21:23, Terence Yim <cht...@gmail.com>: > > > >> Hi Sergey, > >> > >> I think you are talking about TWILL-147 ( > >> https://issues.apache.org/jira/browse/TWILL-147), right? The idea for > that > >> is we don't need to start EmbeddedKafkaServer in AM at all, but rather > it > >> just take a configuration (via TwillPreparer, which can have a default > >> value in the Configuration object passed to YarnTwillRunnerService), > which > >> the configuration specifies the Kafka broker list and topic that the AM > >> will publish to. > >> > >> Since under this model, application logs from different application may > >> send to the same Kafka topic (depends on the configuration), the > LogEntry > >> needs to be modified to carry the application and run id, so that the > >> TwillController can filter based on it on the client side. > >> > >> Terence > >> > >> > >> > >>> On Sat, Jul 1, 2017 at 1:46 AM, Сергей Филиппов <role...@gmail.com> > wrote: > >>> > >>> Hello, > >>> I would like to implement possibility to use external kafka server for > >> log > >>> aggregation. > >>> Now twill uses EmbededKafkaServer for that. I think implementation > would > >>> look like this: > >>> 1. Add ZK path where kafka zk connection string will be stored. There > >>> should should be only one such path per ApplicationMaster > >>> 2. Use this path in ApplicationKafkaService while creating > >>> EmbededKafkaService, if there on brokers right now > >>> 3. For log aggregation there should be additional nodes in ZK for each > >>> instance with kafka topic's name in it. Something like > >>> "test-app-{UUID}-log". So publisher will send to this topic and > consumer > >>> will consume log messages on the job submission machine. > >>> > >>> What would you say? Is this sounds ok? > >>> > >>> Sergey > >>> > >> >
