[jira] [Commented] (TWILL-147) Allow using external Kafka server for log collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107620#comment-16107620
 ] 

ASF GitHub Bot commented on TWILL-147:
--

Github user chtyim commented on the issue:

https://github.com/apache/twill/pull/57
  
One quick question, does Kafka 0.10 client works with older version of 
Kafka server?

Also, ideally we should separate out all these Kafka related dependencies 
in a separate twill module, otherwise making it compatible with different Scala 
version would be quite difficult.


> Allow using external Kafka server for log collection
> 
>
> Key: TWILL-147
> URL: https://issues.apache.org/jira/browse/TWILL-147
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, core
>Reporter: Terence Yim
>Assignee: Filippov Sergey
>
> Show allow user to use any Kafka server for log collection, not necessarily 
> starting an embedded one in the AM.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-147) Allow using external Kafka server for log collection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096014#comment-16096014
 ] 

ASF GitHub Bot commented on TWILL-147:
--

GitHub user fonelor opened a pull request:

https://github.com/apache/twill/pull/57

(TWILL-147) Allow using external Kafka server for log collection

Now external kafka servers could be used for log aggregation
Kafka updated to 0.10.2.0
Embedded kafka server starts if log aggregation is enabled and no external 
kafka servers configured
removed old zk based kafka broker discovery service

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/fonelor/twill feature/kafka_upgrade

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/twill/pull/57.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #57


commit 6816ea9945d76921363960043b0bfc392d320cd9
Author: Sergey Filippov 
Date:   2017-07-14T09:47:21Z

started kafka migration

commit 9f94b4e38ff7bcd8356f9c5671537c9787f93eb6
Author: Sergey Filippov 
Date:   2017-07-18T15:51:47Z

implemented new KafkaConsumer and new KafkaProducer

commit bd4803e8959b632f9c09ddfc4ec644a461175eea
Author: Sergey Filippov 
Date:   2017-07-19T15:20:03Z

fixed jmx problems with client.id for consumer
fixed no payload in logger FetchedMessage
fixed incorrect scheduling and running of KafkaConsumer task

commit e98d2f4f92d15a115ef260095ab2f95b6b990417
Author: Sergey Filippov 
Date:   2017-07-20T15:33:20Z

fixed KafkaTest
removed Encoders
implemented broker.id generation

commit 8bbcb1a417fbeaa22d51b7307c13e1efa3ade71d
Author: Sergey Filippov 
Date:   2017-07-21T08:37:23Z

checkStyle

commit 92937a5b76c368af7ee86f2a495670aaa4d8c6bb
Author: Sergey Filippov 
Date:   2017-07-21T08:50:31Z

fixed compressing setting

commit 71d3db756a79f897c4265443a400375ec9f386bb
Author: Sergey Filippov 
Date:   2017-07-21T09:06:07Z

clean up
revert code moving
added slf4j-log4j12 for testing with kafka

commit 7667767cd54981f55bee12c3cecc69cc5bd00cbd
Author: Sergey Filippov 
Date:   2017-07-21T09:15:20Z

added kafka bootstrap servers configuration to TwillPreparer

commit cbc7a554e43b80a6622f1a065a3501dc27d88e64
Author: Sergey Filippov 
Date:   2017-07-21T09:18:59Z

Removed old kafka services




> Allow using external Kafka server for log collection
> 
>
> Key: TWILL-147
> URL: https://issues.apache.org/jira/browse/TWILL-147
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, core
>Reporter: Terence Yim
>Assignee: Filippov Sergey
>
> Show allow user to use any Kafka server for log collection, not necessarily 
> starting an embedded one in the AM.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-147) Allow using external Kafka server for log collection

[Terence Yim (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089214#comment-16089214
 ] 

Terence Yim commented on TWILL-147:
---

When Twill started using Kafka, Kafka was on version 0.7-0.8, which doesn't 
have a pure java api yet. Things have changed and the general principle is to 
minimize dependencies if it doesn't involve reinventing a lot of code.

Regarding the log appender library, I think the best way to address it is to 
make it somewhat pluggable so that user doesn't need to use the one that comes 
from Twill.

> Allow using external Kafka server for log collection
> 
>
> Key: TWILL-147
> URL: https://issues.apache.org/jira/browse/TWILL-147
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, core
>Reporter: Terence Yim
>
> Show allow user to use any Kafka server for log collection, not necessarily 
> starting an embedded one in the AM.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (TWILL-147) Allow using external Kafka server for log collection

[Filippov Sergey (JIRA)]

[ 
https://issues.apache.org/jira/browse/TWILL-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082156#comment-16082156
 ] 

Filippov Sergey commented on TWILL-147:
---

I tried to migrate from kafka 0.10 to 0.11, and it seems, that a lot of code 
that is in org.apache.twill.kafka.client has "alternatives" in official 
kafka-clients package. Are there any strict restrictions to avoid using 
kafka-clients pakage in twill?
And another thing: there is a nice logback appender specially designed to use 
with kafka, maybe it is better to use it for log aggregation? 
(https://github.com/danielwegener/logback-kafka-appender)

> Allow using external Kafka server for log collection
> 
>
> Key: TWILL-147
> URL: https://issues.apache.org/jira/browse/TWILL-147
> Project: Apache Twill
>  Issue Type: Improvement
>  Components: api, core
>Reporter: Terence Yim
>
> Show allow user to use any Kafka server for log collection, not necessarily 
> starting an embedded one in the AM.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)