[jira] [Commented] (KAFKA-2671) Enable starting Kafka server with a Properties object
[ https://issues.apache.org/jira/browse/KAFKA-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970474#comment-14970474 ] ASF GitHub Bot commented on KAFKA-2671: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/330 > Enable starting Kafka server with a Properties object > - > > Key: KAFKA-2671 > URL: https://issues.apache.org/jira/browse/KAFKA-2671 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Ashish K Singh >Assignee: Ashish K Singh > Fix For: 0.9.0.0 > > > Kafka, as of now, can only be started with a properties file and override > params. It makes life easier for management applications to be able to start > Kafka with a properties object programatically. > The changes required to enable this are minimal, just a tad bit of > refactoring of kafka.Kafka. The changes must maintain current behavior intact. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2671) Enable starting Kafka server with a Properties object
[ https://issues.apache.org/jira/browse/KAFKA-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14967999#comment-14967999 ] Ashish K Singh commented on KAFKA-2671: --- [~gwenshap], [~guozhang] I have updated the patch, could you guys take a look. Thanks! > Enable starting Kafka server with a Properties object > - > > Key: KAFKA-2671 > URL: https://issues.apache.org/jira/browse/KAFKA-2671 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Ashish K Singh >Assignee: Ashish K Singh > > Kafka, as of now, can only be started with a properties file and override > params. It makes life easier for management applications to be able to start > Kafka with a properties object programatically. > The changes required to enable this are minimal, just a tad bit of > refactoring of kafka.Kafka. The changes must maintain current behavior intact. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2671) Enable starting Kafka server with a Properties object
[ https://issues.apache.org/jira/browse/KAFKA-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965456#comment-14965456 ] Ashish K Singh commented on KAFKA-2671: --- [~gwenshap] and [~jkreps] as long as there is an easy way to embed Kafka, my purpose, and many other use cases as mentioned by Jay, will be served. I will update the PR to address that. > Enable starting Kafka server with a Properties object > - > > Key: KAFKA-2671 > URL: https://issues.apache.org/jira/browse/KAFKA-2671 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Ashish K Singh >Assignee: Ashish K Singh > > Kafka, as of now, can only be started with a properties file and override > params. It makes life easier for management applications to be able to start > Kafka with a properties object programatically. > The changes required to enable this are minimal, just a tad bit of > refactoring of kafka.Kafka. The changes must maintain current behavior intact. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2671) Enable starting Kafka server with a Properties object
[ https://issues.apache.org/jira/browse/KAFKA-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964394#comment-14964394 ] Jay Kreps commented on KAFKA-2671: -- The use case for embedding Kafka is to make it work with existing config, monitoring, and deployment tools that require some integration work (logic specific to that framework) and also to support integration testing. Rather than trying to provide plugin hooks for this which are usually kind of limited and hacky you can just have people create their own main method and that way they can add whatever logic on startup or shutdown they want. Plus just as a matter of cleanliness it's nice to be able to have a clean path to do programmatically whatever we're doing in the command line tool. We've tried to support this usage pattern since early on so if it has gotten painful it'd be nice to clean it up a bit. > Enable starting Kafka server with a Properties object > - > > Key: KAFKA-2671 > URL: https://issues.apache.org/jira/browse/KAFKA-2671 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Ashish K Singh >Assignee: Ashish K Singh > > Kafka, as of now, can only be started with a properties file and override > params. It makes life easier for management applications to be able to start > Kafka with a properties object programatically. > The changes required to enable this are minimal, just a tad bit of > refactoring of kafka.Kafka. The changes must maintain current behavior intact. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2671) Enable starting Kafka server with a Properties object
[ https://issues.apache.org/jira/browse/KAFKA-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964360#comment-14964360 ] Gwen Shapira commented on KAFKA-2671: - In second look, metrics are currently handled in both KafkaServer and the Kafka singleton. This looks wrong. Perhaps you can figure out why and see if we can safely move all metrics handling inside KafkaServer? After that, your wrapper will just need to handle the shutdownhook and call KafkaServer, which sounds reasonable to me. > Enable starting Kafka server with a Properties object > - > > Key: KAFKA-2671 > URL: https://issues.apache.org/jira/browse/KAFKA-2671 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Ashish K Singh >Assignee: Ashish K Singh > > Kafka, as of now, can only be started with a properties file and override > params. It makes life easier for management applications to be able to start > Kafka with a properties object programatically. > The changes required to enable this are minimal, just a tad bit of > refactoring of kafka.Kafka. The changes must maintain current behavior intact. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2671) Enable starting Kafka server with a Properties object
[ https://issues.apache.org/jira/browse/KAFKA-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964353#comment-14964353 ] Gwen Shapira commented on KAFKA-2671: - "takes care of config management in a secure way" if there's a security issue with the way Kafka currently manages its configuration, please speak up :) Is this a work-around to loading SSL configuration from an app? > Enable starting Kafka server with a Properties object > - > > Key: KAFKA-2671 > URL: https://issues.apache.org/jira/browse/KAFKA-2671 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Ashish K Singh >Assignee: Ashish K Singh > > Kafka, as of now, can only be started with a properties file and override > params. It makes life easier for management applications to be able to start > Kafka with a properties object programatically. > The changes required to enable this are minimal, just a tad bit of > refactoring of kafka.Kafka. The changes must maintain current behavior intact. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2671) Enable starting Kafka server with a Properties object
[ https://issues.apache.org/jira/browse/KAFKA-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964345#comment-14964345 ] Ashish K Singh commented on KAFKA-2671: --- bq. What's the use-case for starting Kafka server as a Singleton inside another application? I am trying to wrap kafka.Kafka in an app that takes care of config management in a secure way. The goal of the app is not to use Kafka, but to just start it. To be honest its a simple wrapper :). bq. The reason I'm asking is that I've actually seen this requirement before, and it seemed like a rather misguided attempt to use Kafka as a local queue for a webserver (and use a local Kafka rather than a producer to a remote Kafka). I'm rather concerned about enabling such use-cases, as they can lead to more problems down the road. Misguided attempts are still doable with the existing options, right? bq. Also, are you sure you want to instantiate Kafka with a start() method to a singleton? This is kind of non-standard. That is true. Once I get a go on the idea, I can handle that in PR by extracting to a separate class. > Enable starting Kafka server with a Properties object > - > > Key: KAFKA-2671 > URL: https://issues.apache.org/jira/browse/KAFKA-2671 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Ashish K Singh >Assignee: Ashish K Singh > > Kafka, as of now, can only be started with a properties file and override > params. It makes life easier for management applications to be able to start > Kafka with a properties object programatically. > The changes required to enable this are minimal, just a tad bit of > refactoring of kafka.Kafka. The changes must maintain current behavior intact. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2671) Enable starting Kafka server with a Properties object
[ https://issues.apache.org/jira/browse/KAFKA-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964309#comment-14964309 ] Gwen Shapira commented on KAFKA-2671: - What's the use-case for starting Kafka server as a Singleton inside another application? The reason I'm asking is that I've actually seen this requirement before, and it seemed like a rather misguided attempt to use Kafka as a local queue for a webserver (and use a local Kafka rather than a producer to a remote Kafka). I'm rather concerned about enabling such use-cases, as they can lead to more problems down the road. I hope you have a better use-case. Also, are you sure you want to instantiate Kafka with a start() method to a singleton? This is kind of non-standard. > Enable starting Kafka server with a Properties object > - > > Key: KAFKA-2671 > URL: https://issues.apache.org/jira/browse/KAFKA-2671 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Ashish K Singh >Assignee: Ashish K Singh > > Kafka, as of now, can only be started with a properties file and override > params. It makes life easier for management applications to be able to start > Kafka with a properties object programatically. > The changes required to enable this are minimal, just a tad bit of > refactoring of kafka.Kafka. The changes must maintain current behavior intact. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2671) Enable starting Kafka server with a Properties object
[
https://issues.apache.org/jira/browse/KAFKA-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964297#comment-14964297
]
Ashish K Singh commented on KAFKA-2671:
---
Correct me if I am wrong, but below are the options one has to start Kafka
broker programmatically.
1. Pass a config file and overrides.
2. Start Kafka with KafkaConfig, as suggested by Gwen.
Now, consider a scenario where an application has required Kafka properties
available in a Properties object. Following are the ways Kafka can be started
by the app.
1. Create a conf file, and then start Kafka passing the conf file. This seems
to be an overkill. Passing conf file is ideal while starting from CLI, but I am
not sure if that is true when one wants to start programmatically.
2. Create a list of overrides and pass that as an argument while starting
Kafka. Works but is a hack-ish solution.
3. Create KafkaConfig and use that to create Kafka. The app will have to take
care of starting other entities that {{kafka.Kafka}} starts right now, like,
metrics reporter, add a shutdown hook, etc. If tomorrow Kafka decides to start
some other services as well here, application starting Kafka will also have to
take care of these things.
The interface, suggested in the PR, will enable starting Kafka by just passing
a Properties object. Unlike, the 3rd option above, this will be future proof
and app won't have to take care of bunch of other things that {{kafka.Kafka}}
does right now. I agree that this is not a very hard requirement and one can
get around with the options suggested above. Its more for programmers
convenience then anything else, makes Kafka easy to embed in other apps.
> Enable starting Kafka server with a Properties object
> -
>
> Key: KAFKA-2671
> URL: https://issues.apache.org/jira/browse/KAFKA-2671
> Project: Kafka
> Issue Type: Improvement
> Components: core
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> Kafka, as of now, can only be started with a properties file and override
> params. It makes life easier for management applications to be able to start
> Kafka with a properties object programatically.
> The changes required to enable this are minimal, just a tad bit of
> refactoring of kafka.Kafka. The changes must maintain current behavior intact.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2671) Enable starting Kafka server with a Properties object
[ https://issues.apache.org/jira/browse/KAFKA-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964255#comment-14964255 ] Guozhang Wang commented on KAFKA-2671: -- [~singhashish] I am still not clear about the motivation of this patch as [~gwenshap], could you elaborate a bit more? > Enable starting Kafka server with a Properties object > - > > Key: KAFKA-2671 > URL: https://issues.apache.org/jira/browse/KAFKA-2671 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Ashish K Singh >Assignee: Ashish K Singh > > Kafka, as of now, can only be started with a properties file and override > params. It makes life easier for management applications to be able to start > Kafka with a properties object programatically. > The changes required to enable this are minimal, just a tad bit of > refactoring of kafka.Kafka. The changes must maintain current behavior intact. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2671) Enable starting Kafka server with a Properties object
[ https://issues.apache.org/jira/browse/KAFKA-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964017#comment-14964017 ] Ashish K Singh commented on KAFKA-2671: --- [~gwenshap], yes it is doable, however one would have to start metrics reporter, kafka server, add a shutdown hook, etc. If tomorrow Kafka decides to start some other services as well here, application starting Kafka will have to take care of these things. > Enable starting Kafka server with a Properties object > - > > Key: KAFKA-2671 > URL: https://issues.apache.org/jira/browse/KAFKA-2671 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Ashish K Singh >Assignee: Ashish K Singh > > Kafka, as of now, can only be started with a properties file and override > params. It makes life easier for management applications to be able to start > Kafka with a properties object programatically. > The changes required to enable this are minimal, just a tad bit of > refactoring of kafka.Kafka. The changes must maintain current behavior intact. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2671) Enable starting Kafka server with a Properties object
[ https://issues.apache.org/jira/browse/KAFKA-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14963997#comment-14963997 ] Gwen Shapira commented on KAFKA-2671: - You can start Kafka with KafkaConfig object (I believe LinkedIn are doing this). Since you can create KafkaConfig from properties object, I'm not sure why this additional interface is needed. > Enable starting Kafka server with a Properties object > - > > Key: KAFKA-2671 > URL: https://issues.apache.org/jira/browse/KAFKA-2671 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Ashish K Singh >Assignee: Ashish K Singh > > Kafka, as of now, can only be started with a properties file and override > params. It makes life easier for management applications to be able to start > Kafka with a properties object programatically. > The changes required to enable this are minimal, just a tad bit of > refactoring of kafka.Kafka. The changes must maintain current behavior intact. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2671) Enable starting Kafka server with a Properties object
[ https://issues.apache.org/jira/browse/KAFKA-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14963985#comment-14963985 ] ASF GitHub Bot commented on KAFKA-2671: --- GitHub user SinghAsDev opened a pull request: https://github.com/apache/kafka/pull/330 KAFKA-2671: Enable starting Kafka server with a Properties object You can merge this pull request into a Git repository by running: $ git pull https://github.com/SinghAsDev/kafka KAFKA-2671 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/330.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #330 commit 3924b1da522f257bc45d51b98af247208dfc210c Author: Ashish Singh Date: 2015-10-19T20:21:08Z KAFKA-2671: Enable starting Kafka server with a Properties object > Enable starting Kafka server with a Properties object > - > > Key: KAFKA-2671 > URL: https://issues.apache.org/jira/browse/KAFKA-2671 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Ashish K Singh >Assignee: Ashish K Singh > > Kafka, as of now, can only be started with a properties file and override > params. It makes life easier for management applications to be able to start > Kafka with a properties object programatically. > The changes required to enable this are minimal, just a tad bit of > refactoring of kafka.Kafka. The changes must maintain current behavior intact. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
