+ kafka-dev On Fri, Aug 5, 2011 at 10:14 AM, Jay Kreps <jay.kr...@gmail.com> wrote:
> Actually I am not really sure. What the deploy thing was doing was the ec2 > allocation stuff in addition to the rsync, which I am quite sure duplicates > whirr. If whirr is also easy to use on your own (non-ec2) machines (or ec2 > machines you have already allocated) then that maybe that is a preferable > approach for everything and we should just write up a recipe for how to use > whirr with kafka. > > Anyhow feel free to treat it as a separate issue, I just thought i would > throw it out there since i think that kind of "out of the box experience" is > really important for people to be able to get up and running before they > really commit to something. > > -Jay > > > On Fri, Aug 5, 2011 at 6:14 AM, Chris Burroughs <chris.burrou...@gmail.com > > wrote: > >> All sounds cool and doable. But didn't we delete contrib/deploy in >> favour of https://issues.apache.org/jira/browse/KAFKA-10 ? >> >> I agree that something like https://github.com/pcmanus/ccm would be >> useful for bringing up local clusters for local testing. >> >> (And yeah, we should create a pid file instead of killing every running >> kafka) >> >> On 08/04/2011 09:17 PM, Jay Kreps wrote: >> > One thing I think would be really helpful would be to make it so you >> could >> > easily spawn up a cluster. Hadoop does this and it is nice. For >> companies >> > that don't have standardized deployment infrastructure this is really >> nice, >> > and also for testing of all kinds. Basically it would be nice if you >> could >> > make a list containing one host:port per line and there were scripts >> like >> > bin/kafka-cluster-deploy.sh machine-list.txt # rsync the kafka >> > directory and config around to the given list of machines >> > bin/kafka-cluster-start.sh machine-list.txt # ssh to each >> machine >> > in the list and start it >> > bin/kafka-cluster-stop.sh machine-list.txt # ssh to each >> machine >> > and kill the kafka process >> > bin/kafka-cluster-delete.sh machine-list.txt # ssh around to each >> > machine in the list and delete the code and log directory >> > >> > To do this I think you would need a way to override properties on the >> > command line. This would make it so you could rsync out the kafka code >> to a >> > bunch of machines but give a different node id and (if needed) port. I >> think >> > something like "--property-name value" to give properties that override >> what >> > is in the properties file would work. >> > >> > One other minor and unrelated issue we have is that our stop script >> actually >> > kills all the kafka processes on the machine, which means if you are >> trying >> > to run multiple nodes on the same machine it is a little dangerous. >> > >> > -Jay >> > >> > On Thu, Aug 4, 2011 at 4:59 PM, Chris Burroughs >> > <chris.burrou...@gmail.com>wrote: >> > >> >> I've been looking at how kafka is packaged and the provided scripts for >> >> running it. There are a few things I want to improve: it should be >> >> easier to run in the foreground or background, convincing java to keep >> >> the executable bits set, and reasonable log4j defaults. There are also >> >> a few plain old bugs detailed in KAFKA-81 that need to be fixed. >> >> >> >> If there are other things that would make the results of ./sbt package >> >> or release-zip more useful, or the bash scripts easier to work with, >> >> please let me know. >> >> >> >> >> >> Thanks, >> >> Chris Burroughs >> >> >> > >> >> >
