[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15206680#comment-15206680 ] Stefan Gower commented on KAFKA-1173: - Thanks. Will retry. Stefan From: Ewen Cheslack-Postava (JIRA) To: ste...@topicscout.com Sent: Sunday, March 20, 2016 3:50 PM Subject: [jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka [ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203558#comment-15203558 ] Ewen Cheslack-Postava commented on KAFKA-1173: -- [~sfgower] Those instructions are out of date. The version that was ultimately checked in has instructions in vagrant/README.md -- This message was sent by Atlassian JIRA (v6.3.4#6332) > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.9.0.0 > > Attachments: KAFKA-1173-JMX.patch, KAFKA-1173.patch, > KAFKA-1173_2013-12-07_12:07:55.patch, KAFKA-1173_2014-11-11_13:50:55.patch, > KAFKA-1173_2014-11-12_11:32:09.patch, KAFKA-1173_2014-11-18_16:01:33.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
I had good experiences using the vagrant setup as it is on a mac, but did had to change some things. We are using docker now. I'm not sure about the general preference, but I would like a docker compose over the vagrant setup. Don't know if you really want it Kafka itself, and to give it support through. On Mon, Mar 21, 2016 at 9:43 PM Ewen Cheslack-Postava (JIRA) < j...@apache.org> wrote: > > [ > https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205083#comment-15205083 > ] > > Ewen Cheslack-Postava commented on KAFKA-1173: > -- > > [~gwenshap] Maybe? I had been thinking of our Vagrantfile as a tool for > Kafka developers. Technically I guess it gets shipped with the source > version. It doesn't get shipped with the binary versions afaik, which may > be confusing. > > I guess it's also a question of whether we want to treat it as > "supported"... > > > Using Vagrant to get up and running with Apache Kafka > > - > > > > Key: KAFKA-1173 > > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > > Project: Kafka > > Issue Type: Improvement > >Reporter: Joe Stein > >Assignee: Ewen Cheslack-Postava > > Fix For: 0.9.0.0 > > > > Attachments: KAFKA-1173-JMX.patch, KAFKA-1173.patch, > KAFKA-1173_2013-12-07_12:07:55.patch, KAFKA-1173_2014-11-11_13:50:55.patch, > KAFKA-1173_2014-11-12_11:32:09.patch, KAFKA-1173_2014-11-18_16:01:33.patch > > > > > > Vagrant has been getting a lot of pickup in the tech communities. I > have found it very useful for development and testing and working with a > few clients now using it to help virtualize their environments in > repeatable ways. > > Using Vagrant to get up and running. > > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > > 1) Install Vagrant [ > http://www.vagrantup.com/](http://www.vagrantup.com/) > > 2) Install Virtual Box [ > https://www.virtualbox.org/](https://www.virtualbox.org/) > > In the main kafka folder > > 1) ./sbt update > > 2) ./sbt package > > 3) ./sbt assembly-package-dependency > > 4) vagrant up > > once this is done > > * Zookeeper will be running 192.168.50.5 > > * Broker 1 on 192.168.50.10 > > * Broker 2 on 192.168.50.20 > > * Broker 3 on 192.168.50.30 > > When you are all up and running you will be back at a command brompt. > > If you want you can login to the machines using vagrant shh > but you don't need to. > > You can access the brokers and zookeeper by their IP > > e.g. > > bin/kafka-console-producer.sh --broker-list 192.168.50.10:9092, > 192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic > sandbox --from-beginning > > > > -- > This message was sent by Atlassian JIRA > (v6.3.4#6332) >
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205083#comment-15205083 ] Ewen Cheslack-Postava commented on KAFKA-1173: -- [~gwenshap] Maybe? I had been thinking of our Vagrantfile as a tool for Kafka developers. Technically I guess it gets shipped with the source version. It doesn't get shipped with the binary versions afaik, which may be confusing. I guess it's also a question of whether we want to treat it as "supported"... > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.9.0.0 > > Attachments: KAFKA-1173-JMX.patch, KAFKA-1173.patch, > KAFKA-1173_2013-12-07_12:07:55.patch, KAFKA-1173_2014-11-11_13:50:55.patch, > KAFKA-1173_2014-11-12_11:32:09.patch, KAFKA-1173_2014-11-18_16:01:33.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203751#comment-15203751 ] Gwen Shapira commented on KAFKA-1173: - [~ewencp] Does it make sense to copy vagrant instructions to our documentation? I think it will increase visibility for searches. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.9.0.0 > > Attachments: KAFKA-1173-JMX.patch, KAFKA-1173.patch, > KAFKA-1173_2013-12-07_12:07:55.patch, KAFKA-1173_2014-11-11_13:50:55.patch, > KAFKA-1173_2014-11-12_11:32:09.patch, KAFKA-1173_2014-11-18_16:01:33.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203558#comment-15203558 ] Ewen Cheslack-Postava commented on KAFKA-1173: -- [~sfgower] Those instructions are out of date. The version that was ultimately checked in has instructions in vagrant/README.md > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.9.0.0 > > Attachments: KAFKA-1173-JMX.patch, KAFKA-1173.patch, > KAFKA-1173_2013-12-07_12:07:55.patch, KAFKA-1173_2014-11-11_13:50:55.patch, > KAFKA-1173_2014-11-12_11:32:09.patch, KAFKA-1173_2014-11-18_16:01:33.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202404#comment-15202404 ] Stefan Gower commented on KAFKA-1173: - It is certainly helpful to have a way to quickly set up a virtual cluster running Kafka. Thank! So I tried your installation instructions on a Mac (running El Capitan). An error occurred on one of the final steps.. Per the instructions, this command was invoked in the kafka directory. sbt assembly-package-dependency [info] Set current project to kafka (in build file:/Users/TRINITY/vagrants/tmp/kafka/) [error] Not a valid command: assembly-package-dependency [error] Not a valid project ID: assembly-package-dependency [error] Expected ':' (if selecting a configuration) [error] Not a valid key: assembly-package-dependency (similar: sbt-dependency) [error] assembly-package-dependency [error]^ > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.9.0.0 > > Attachments: KAFKA-1173-JMX.patch, KAFKA-1173.patch, > KAFKA-1173_2013-12-07_12:07:55.patch, KAFKA-1173_2014-11-11_13:50:55.patch, > KAFKA-1173_2014-11-12_11:32:09.patch, KAFKA-1173_2014-11-18_16:01:33.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605003#comment-14605003 ] Joe Stein commented on KAFKA-1173: -- Can you create a new ticket please, thanks. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > Attachments: KAFKA-1173-JMX.patch, KAFKA-1173.patch, > KAFKA-1173_2013-12-07_12:07:55.patch, KAFKA-1173_2014-11-11_13:50:55.patch, > KAFKA-1173_2014-11-12_11:32:09.patch, KAFKA-1173_2014-11-18_16:01:33.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234816#comment-14234816 ] Joe Stein commented on KAFKA-1173: -- [~ewencp] fell off my radar some will circle back on this next day or so and catch back up > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch, > KAFKA-1173_2014-11-18_16:01:33.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14233551#comment-14233551 ] Ewen Cheslack-Postava commented on KAFKA-1173: -- [~joestein] Just wanted to ping you on this, any progress with testing? > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch, > KAFKA-1173_2014-11-18_16:01:33.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1469#comment-1469 ] Neha Narkhede commented on KAFKA-1173: -- +1 on making this such that a simple quick start and producer/consumer tests are easy to run from outside the VPC. The way I see this is making it easy for folks to get started with Kafka out of the box. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch, > KAFKA-1173_2014-11-18_16:01:33.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14221118#comment-14221118 ] Joe Stein commented on KAFKA-1173: -- [~ewencp] I think we both agree we are not trying to deal with every use case just making sure folks don't have a bad experience. Let me go through your latest patch and should be in a place I can commit when i get some time for spin ups/down and such again. Thanks! > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch, > KAFKA-1173_2014-11-18_16:01:33.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14217099#comment-14217099 ] Ewen Cheslack-Postava commented on KAFKA-1173: -- [~joestein] I fixed the issue where it was using the static IP addresses intended for local VMs by only assigning them under Virtualbox. Unfortunately, that doesn't help with your problem. The vagrant-aws plugin searches for addresses to use in a fixed order (https://github.com/mitchellh/vagrant-aws/blob/master/lib/vagrant-aws/action/read_ssh_info.rb#L36) and the SSH config is all it exposes. For that it obviously makes sense for them to prefer the public address. The vagrant-hostmanager plugin only looks in a couple of places for addresses to use (https://github.com/smdahlen/vagrant-hostmanager/blob/master/lib/vagrant-hostmanager/hosts_file/updater.rb#L101), the final fallback being the SSH config. As it stands, there's no way to get the private IP out of the vagrant-aws plugin and into the vagrant-hostmanager plugin. At this point I think it will require a patch to at least vagrant-aws to make it work with your VPC config. We might be able to make it work for a pure VPC setup where you're running Vagrant in the VPC since vagrant-aws wouldn't be able to use public addresses in that case, but I doubt that would be a common use case. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch, > KAFKA-1173_2014-11-18_16:01:33.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14217083#comment-14217083 ] Ewen Cheslack-Postava commented on KAFKA-1173: -- Updated reviewboard https://reviews.apache.org/r/27735/diff/ against branch origin/trunk > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch, > KAFKA-1173_2014-11-18_16:01:33.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14215675#comment-14215675 ] Joe Stein commented on KAFKA-1173: -- [~ewencp] changing override.hostmanager.ignore_private_ip = false in the AWS section didn't work :( The host manager is setting the hosts to the 192 address cat /etc/hosts ## vagrant-hostmanager-start 192.168.50.51 broker1 192.168.50.52 broker2 192.168.50.53 broker3 192.168.50.11 zk1 The virtual box parts are great I think for folks to jump in and get up and running quickly using vagrant and it is helpful for development and works without futzing with it, yup. One option is we could commit that part and move the AWS pieces to another ticket. I don't mind that but I am ok with helping to keep testing the EC2 parts as long as it can work for folks out the box with little issues/steps as our end game. I should have a chance to try this all again and/or review whatever changes on Wednesday & Thursdasy (FYI gotta knock off for the evening and tomorrow is packed). Many folks have VPC we should try to accommodate them otherwise it just looks like Kafka isn't working (or is harder than it really is to setup). We already get a lot of emails about EC2 and advertising hosts and everything else so this could be really helpful for folks once it is working more. << The Spark EC2 scripts do a nice job of just setting up a usable default so it's really easy to get up and running, but I'm also hesitant to have the script automatically muck with users' security settings. You can make it a flag to use it and have some detail about changing the flag and what is going to happen if you do (so it is not automatic). I think if things are going to work then folks can make the decision themselves if the impact of that is something that is worth it for them. I think having it just not work though without having to-do a lot kind of takes away from the "spin up something working" aspect to the change. We will also find out a lot more and learn what the community wants more and/or differently as this gets in and out to the world. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14215661#comment-14215661 ] Ewen Cheslack-Postava commented on KAFKA-1173: -- [~joestein] Got it, makes sense. The EC2 security group thing is a pain, it ends up requiring too much manual effort. The Spark EC2 scripts do a nice job of just setting up a usable default so it's really easy to get up and running, but I'm also hesitant to have the script automatically muck with users' security settings. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14215653#comment-14215653 ] Joe Stein commented on KAFKA-1173: -- [~ewencp] I agree no more toggles unless required. I am looking at this from the community perspective and thinking that once we commit this we have to support it and all the questions/issues people are going to have and will come over the mailing list (etc). I flipped ignore_private_ip to false in the aws section of the vagrant file and giving it another try. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14215639#comment-14215639 ] Ewen Cheslack-Postava commented on KAFKA-1173: -- You need ec2_associate_public_ip if you want to be able to SSH externally. I think for your setup you'd want to set override.hostmanager.ignore_private_ip = false, whereas I specifically set it to true to get the public address. You'd only want to turn of ec2_associate_public_ip if everything including vagrant was running inside your VPC, e.g. internal automated tests that leverage the Vagrant script to setup the cluster. To explain why I wrote things the way I did -- my primary use case is development of and on top of Kafka. I want to make it easy to setup the cluster, run admin commands to inspect its state, setup producer/consumer processes, and, when necessary, SSH in and debug things. A lot of that can be done from my laptop, so supporting access from outside the VPC is handy. My setup is definitely not secure, but that's kind of by design -- if I'm just dumping test data into the cluster, I'm not particularly concerned about the security of the data. (But it would suck if someone else randomly started publishing data to my cluster...). I'm hesitant of adding even more toggles -- eventually it gets so complex that its easier for each person to write their own custom Vagrantfile. And the amount of effort to get up and running on EC2 is already pretty high. Thoughts on a good compromise? The primary use cases I was thinking about were Kafka development (i.e. patch needs testing against a real cluster, system tests are breaking and I need to reproduce the issue), demo/tutorial (i.e. help users get a real cluster they can test against up and running), and a testbed for application-level code and benchmarks. It sounds like you either have a slightly different use case or just have a different workflow for using EC2 during development. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14215589#comment-14215589 ] Joe Stein commented on KAFKA-1173: -- [~ewencp] The aws parts are further along and servers are spinning up with the code and configs installed but still getting issue. All of the DNS hosts are on the public IP but my default security group is only setup for 22 on the outside and only the internal security group for inside. Can this change to use the private IP instead of the public address? should I have set ec2_associate_public_ip = false I ran vagrant outside the VPC but maybe I should change it to false though the comment above that line is the reason I didn't? > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14215523#comment-14215523 ] Joe Stein commented on KAFKA-1173: -- ignore last issue I quoted everything and spinning up now > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14215515#comment-14215515 ] Joe Stein commented on KAFKA-1173: -- FYI - trying a new AWS secret key but got error because I had a "+" in my key that AWS generated There is a syntax error in the following Vagrantfile. The syntax error message is reproduced below for convenience: Vagrantfile.local:3: syntax error, unexpected tIDENTIFIER, expecting end-of-input I am going to generate a new key and continue to test > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14214748#comment-14214748 ] Joe Stein commented on KAFKA-1173: -- I haven't had a chance to review the last changes for issues found with AWS but hopefully will do that today. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14214296#comment-14214296 ] Manikumar Reddy commented on KAFKA-1173: +1 for this patch. It is really useful for creating development test beds. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208515#comment-14208515 ] Ewen Cheslack-Postava commented on KAFKA-1173: -- [~joestein] To be honest, I'm not too surprised something is coming up with the EC2 support. In theory it should be simple, but VPCs introduce a bunch of variables, and testing is tricky since some defaults depend on the age of your account since that affects whether you have EC2 classic support. I ran through a test with a VPC and found some issues. I updated the patch, including some additional info in the README since setting up under a VPC requires slight differences. My testing so far has been in EC2-Classic since that's the default for my account. I also put this VPC in a different region to make sure that worked. Finally, I've noticed that the default parallel provisioning seems to work fine until the very end, when it sometimes seems to hang. I couldn't easily track down the cause, so I updated the readme to use --no-parallel when using EC2. Not ideal, but it works reliably until we can find a better fix. Hopefully those fixes will clear up the issue you're seeing. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208512#comment-14208512 ] Ewen Cheslack-Postava commented on KAFKA-1173: -- Updated reviewboard https://reviews.apache.org/r/27735/diff/ against branch origin/trunk > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208281#comment-14208281 ] Joe Stein commented on KAFKA-1173: -- [~ewencp] I am +1 on the virtual box parts to the patch and the updates you last made (I really like how you added the vagrant part to the main README, nice touch. I am having issue with the EC2 pieces but am pretty convinced it is my account how it is setup with VPC so I am going to setup a new account and try it again. I may not have a chance to-do that until the weekend FYI. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207148#comment-14207148 ] Ewen Cheslack-Postava commented on KAFKA-1173: -- Updated reviewboard https://reviews.apache.org/r/27735/diff/ against branch origin/trunk > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, > KAFKA-1173_2014-11-11_13:50:55.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14205952#comment-14205952 ] Joe Stein commented on KAFKA-1173: -- This looks really great, so far only minor things (started updating the reviewboard) with the virtualbox provider. I didn't realize you were using Vagrantfile.local for the local params so what i was saying about the ENV or AWS setting probably is invalid since Vagrantfile.local is in the .gitignore. It might make sense to call that out in the README "If you want to set local variables create a file called Vagrantfile.local and set them in there" or something. The AWS provider is either being cranky or maybe I am doing something wrong will try again tomorrow when less tired. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein >Assignee: Ewen Cheslack-Postava > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204807#comment-14204807 ] Joe Stein commented on KAFKA-1173: -- I should have a chance to give the script a spin up/down and review over the next couple days. I did a quick scan through the patch and like the premise of how it is done and geared to be used. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14202396#comment-14202396 ] Ewen Cheslack-Postava commented on KAFKA-1173: -- I took [~joestein]'s patch and built it out a bit. Here's what I added: * Add EC2 support if you install the vagrant-aws plugin. * Uses vagrant-hostmanager to manage VM hostnames and configure /etc/hosts. This gives the machines names like zk1, broker2. It can optionally configure the host machines /etc/hosts as well, which is particularly nice when you're dealing with EC2 or running a producer/consumer process on the host. * Optionally uses vagrant-cachier to avoid duplicated downloads across VMs. Works for the normal things vagrant-cachier normally supports (apt-get, rubygems, etc.) as well as specifically for the JDK setup, which requires some customized code because it's downloaded from Oracle. * Local config support via Vagrantfile.local (which is just interpreted as Ruby code). Lets you easily override settings without accidentally checking in those changes. * Provide 3 types of machines: zookeeper, broker, and "worker", where workers just have the code synced but don't start any services. These are useful for running other processes (producers, consumers, admin operations) or you can use only worker nodes to run system tests. * Supports zookeeper ensembles instead of just a single Zookeepr node. Limitations: * Scripts are still only setup for Debian/Ubuntu systems. They could easily be generalized. * I haven't figured out the best way to handle memory settings. They're just fixed at a bit above 1G right now because of the settings in the scripts that start services. * vagrant-hostmanager is now a hard dependency. Unfortunate, but it seems like the best solution if the Vagrantfile is going to work with any Vagrant providers besides local VMs. * Some default values need ongoing maintenance (default EC2 AMI setting, for example) Having a Vagrantfile is nice for testing purposes and a patch like this is low risk since it's not touching any of the main code. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14202395#comment-14202395 ] Ewen Cheslack-Postava commented on KAFKA-1173: -- Created reviewboard https://reviews.apache.org/r/27735/diff/ against branch origin/trunk > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13888705#comment-13888705 ] Neha Narkhede commented on KAFKA-1173: -- Not sure what needs to happen here, moving out of 0.8.1 > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein > Attachments: KAFKA-1173_2013-12-07_12:07:55.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850631#comment-13850631 ] Joe Stein commented on KAFKA-1173: -- I am ok with working it over at https://github.com/stealthly/ but was thinking it could be really helpful for development. I was hoping it would be used (how I would use it personally) to help review and test patches e.g. if this patch was on trunk, then I could patch local the gradle JIRA, do commands for new build and then vagrant up and do more testing (instead of running any sbt commands to try to build server, jars, etc). when working on fixes, could do same before running review tool it could then also help (with supporting projects) test out releases and all the different builds and jars as working too, and more. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein > Fix For: 0.8.1 > > Attachments: KAFKA-1173_2013-12-07_12:07:55.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13843326#comment-13843326 ] Jun Rao commented on KAFKA-1173: Joe, Thanks for the patch. Do you think it's better to keep this in the Apache repo or somewhere else and just link to it in our wiki? > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein > Fix For: 0.8.1 > > Attachments: KAFKA-1173_2013-12-07_12:07:55.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13842833#comment-13842833 ] Kostya Golikov commented on KAFKA-1173: --- I think it is important to notice somewhere that this path will only target debian-based systems. > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein > Fix For: 0.8.1 > > Attachments: KAFKA-1173_2013-12-07_12:07:55.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13842266#comment-13842266 ] Joe Stein commented on KAFKA-1173: -- Updated reviewboard https://reviews.apache.org/r/16101/ against branch origin/trunk > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein > Fix For: 0.8.1 > > Attachments: KAFKA-1173_2013-12-07_12:07:55.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13842263#comment-13842263 ] Joe Stein commented on KAFKA-1173: -- Updated reviewboard https://reviews.apache.org/r/16101/ against branch origin/trunk > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein > Fix For: 0.8.1 > > Attachments: KAFKA-1173_2013-12-07_12:01:25.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13842166#comment-13842166 ] Joe Stein commented on KAFKA-1173: -- Updated reviewboard https://reviews.apache.org/r/16101/ against branch origin/trunk > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein > Fix For: 0.8.1 > > Attachments: KAFKA-1173_2013-12-07_04:57:10.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13842164#comment-13842164 ] Joe Stein commented on KAFKA-1173: -- Updated reviewboard https://reviews.apache.org/r/16101/ against branch origin/trunk > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein > Fix For: 0.8.1 > > Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_04:53:52.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13842146#comment-13842146 ] Joe Stein commented on KAFKA-1173: -- Created reviewboard https://reviews.apache.org/r/16101/ against branch origin/trunk > Using Vagrant to get up and running with Apache Kafka > - > > Key: KAFKA-1173 > URL: https://issues.apache.org/jira/browse/KAFKA-1173 > Project: Kafka > Issue Type: Improvement >Reporter: Joe Stein > Fix For: 0.8.1 > > Attachments: KAFKA-1173.patch > > > Vagrant has been getting a lot of pickup in the tech communities. I have > found it very useful for development and testing and working with a few > clients now using it to help virtualize their environments in repeatable ways. > Using Vagrant to get up and running. > For 0.8.0 I have a patch on github https://github.com/stealthly/kafka > 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/) > 2) Install Virtual Box > [https://www.virtualbox.org/](https://www.virtualbox.org/) > In the main kafka folder > 1) ./sbt update > 2) ./sbt package > 3) ./sbt assembly-package-dependency > 4) vagrant up > once this is done > * Zookeeper will be running 192.168.50.5 > * Broker 1 on 192.168.50.10 > * Broker 2 on 192.168.50.20 > * Broker 3 on 192.168.50.30 > When you are all up and running you will be back at a command brompt. > If you want you can login to the machines using vagrant shh but > you don't need to. > You can access the brokers and zookeeper by their IP > e.g. > bin/kafka-console-producer.sh --broker-list > 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox > bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox > --from-beginning -- This message was sent by Atlassian JIRA (v6.1#6144)
