[jira] [Commented] (TWILL-147) Allow using external Kafka server for log collection
[ https://issues.apache.org/jira/browse/TWILL-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107620#comment-16107620 ] ASF GitHub Bot commented on TWILL-147: -- Github user chtyim commented on the issue: https://github.com/apache/twill/pull/57 One quick question, does Kafka 0.10 client works with older version of Kafka server? Also, ideally we should separate out all these Kafka related dependencies in a separate twill module, otherwise making it compatible with different Scala version would be quite difficult. > Allow using external Kafka server for log collection > > > Key: TWILL-147 > URL: https://issues.apache.org/jira/browse/TWILL-147 > Project: Apache Twill > Issue Type: Improvement > Components: api, core >Reporter: Terence Yim >Assignee: Filippov Sergey > > Show allow user to use any Kafka server for log collection, not necessarily > starting an embedded one in the AM. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TWILL-147) Allow using external Kafka server for log collection
[ https://issues.apache.org/jira/browse/TWILL-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096014#comment-16096014 ] ASF GitHub Bot commented on TWILL-147: -- GitHub user fonelor opened a pull request: https://github.com/apache/twill/pull/57 (TWILL-147) Allow using external Kafka server for log collection Now external kafka servers could be used for log aggregation Kafka updated to 0.10.2.0 Embedded kafka server starts if log aggregation is enabled and no external kafka servers configured removed old zk based kafka broker discovery service You can merge this pull request into a Git repository by running: $ git pull https://github.com/fonelor/twill feature/kafka_upgrade Alternatively you can review and apply these changes as the patch at: https://github.com/apache/twill/pull/57.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #57 commit 6816ea9945d76921363960043b0bfc392d320cd9 Author: Sergey Filippov Date: 2017-07-14T09:47:21Z started kafka migration commit 9f94b4e38ff7bcd8356f9c5671537c9787f93eb6 Author: Sergey Filippov Date: 2017-07-18T15:51:47Z implemented new KafkaConsumer and new KafkaProducer commit bd4803e8959b632f9c09ddfc4ec644a461175eea Author: Sergey Filippov Date: 2017-07-19T15:20:03Z fixed jmx problems with client.id for consumer fixed no payload in logger FetchedMessage fixed incorrect scheduling and running of KafkaConsumer task commit e98d2f4f92d15a115ef260095ab2f95b6b990417 Author: Sergey Filippov Date: 2017-07-20T15:33:20Z fixed KafkaTest removed Encoders implemented broker.id generation commit 8bbcb1a417fbeaa22d51b7307c13e1efa3ade71d Author: Sergey Filippov Date: 2017-07-21T08:37:23Z checkStyle commit 92937a5b76c368af7ee86f2a495670aaa4d8c6bb Author: Sergey Filippov Date: 2017-07-21T08:50:31Z fixed compressing setting commit 71d3db756a79f897c4265443a400375ec9f386bb Author: Sergey Filippov Date: 2017-07-21T09:06:07Z clean up revert code moving added slf4j-log4j12 for testing with kafka commit 7667767cd54981f55bee12c3cecc69cc5bd00cbd Author: Sergey Filippov Date: 2017-07-21T09:15:20Z added kafka bootstrap servers configuration to TwillPreparer commit cbc7a554e43b80a6622f1a065a3501dc27d88e64 Author: Sergey Filippov Date: 2017-07-21T09:18:59Z Removed old kafka services > Allow using external Kafka server for log collection > > > Key: TWILL-147 > URL: https://issues.apache.org/jira/browse/TWILL-147 > Project: Apache Twill > Issue Type: Improvement > Components: api, core >Reporter: Terence Yim >Assignee: Filippov Sergey > > Show allow user to use any Kafka server for log collection, not necessarily > starting an embedded one in the AM. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TWILL-147) Allow using external Kafka server for log collection
[ https://issues.apache.org/jira/browse/TWILL-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089214#comment-16089214 ] Terence Yim commented on TWILL-147: --- When Twill started using Kafka, Kafka was on version 0.7-0.8, which doesn't have a pure java api yet. Things have changed and the general principle is to minimize dependencies if it doesn't involve reinventing a lot of code. Regarding the log appender library, I think the best way to address it is to make it somewhat pluggable so that user doesn't need to use the one that comes from Twill. > Allow using external Kafka server for log collection > > > Key: TWILL-147 > URL: https://issues.apache.org/jira/browse/TWILL-147 > Project: Apache Twill > Issue Type: Improvement > Components: api, core >Reporter: Terence Yim > > Show allow user to use any Kafka server for log collection, not necessarily > starting an embedded one in the AM. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (TWILL-147) Allow using external Kafka server for log collection
[ https://issues.apache.org/jira/browse/TWILL-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082156#comment-16082156 ] Filippov Sergey commented on TWILL-147: --- I tried to migrate from kafka 0.10 to 0.11, and it seems, that a lot of code that is in org.apache.twill.kafka.client has "alternatives" in official kafka-clients package. Are there any strict restrictions to avoid using kafka-clients pakage in twill? And another thing: there is a nice logback appender specially designed to use with kafka, maybe it is better to use it for log aggregation? (https://github.com/danielwegener/logback-kafka-appender) > Allow using external Kafka server for log collection > > > Key: TWILL-147 > URL: https://issues.apache.org/jira/browse/TWILL-147 > Project: Apache Twill > Issue Type: Improvement > Components: api, core >Reporter: Terence Yim > > Show allow user to use any Kafka server for log collection, not necessarily > starting an embedded one in the AM. -- This message was sent by Atlassian JIRA (v6.4.14#64029)