Re: Pros and cons of dockerizing kafka brokers?
blockquote, div.yahoo_quoted { margin-left: 0 !important; border-left:1px
#715FFA solid !important; padding-left:1ex !important; background-color:white
!important; } i am running 5 broker kafka cluster on docker and using mesos
cluster manager and marathon framework
Sent from Yahoo Mail for iPhone
On Saturday, July 9, 2016, 00:00, Christian wrote:
We're using AWS ECS for our Kafka cluster of six nodes. We did some
performance testing on a three node cluster and the results were as good as
the Linkedin published results on bare metal machines.
We are using EBS st1 drives. The bottleneck is the network to the ebs
volumes. So for about 25% more cost, we doubled our vms, using twice as
many 1/2 sized EBS volumes.
-Christian
On Fri, Jul 8, 2016 at 12:07 PM Krish wrote:
> Thanks, Christian.
> I am currently reading about kafka-on-mesos.
> I will hack something this weekend to see if I can bring up a kafka
> scheduler on mesos using dockerized brokers. .
>
>
>
> --
> κρισhναν
>
> On Thu, Jul 7, 2016 at 7:29 PM, Christian Posta >
> wrote:
>
> > One thing I can think of is Kafka likes lots of OS page cache.
> Dockerizing
> > from the standpoint of packaging configs is a good idea, just make sure
> if
> > you're running many brokers together on the same host, they've got enough
> > resources (CPU/Mem) so they don't starve each other.
> >
> > On Thu, Jul 7, 2016 at 2:30 AM, Krish wrote:
> >
> >> Hi,
> >> I am currently testing a custom docker volume driver plugin for AWS
> >> EFS/EBS
> >> access and mounting. So, running kafka broker inside a container makes
> >> will
> >> ease up a lot of configuration issues wrt storage for me.
> >>
> >> Are there any pros and cons of dockerizing kafka broker?
> >> Off the top of my head, since kafka forms the base of our setup, I can
> >> think of making is use the host networking stack, and increase ulimits
> for
> >> the container.
> >> I would like to know if and when kafka becomes greedy and cannibalizes
> >> resources; I can also ensure that it runs on a dedicated machine.
> >>
> >> Thanks.
> >>
> >> Best,
> >> Krish
> >>
> >
> >
> >
> > --
> > *Christian Posta*
> > twitter: @christianposta
> > http://www.christianposta.com/blog
> > http://fabric8.io
> >
> >
>
Re: Pros and cons of dockerizing kafka brokers?
We're using AWS ECS for our Kafka cluster of six nodes. We did some performance testing on a three node cluster and the results were as good as the Linkedin published results on bare metal machines. We are using EBS st1 drives. The bottleneck is the network to the ebs volumes. So for about 25% more cost, we doubled our vms, using twice as many 1/2 sized EBS volumes. -Christian On Fri, Jul 8, 2016 at 12:07 PM Krish wrote: > Thanks, Christian. > I am currently reading about kafka-on-mesos. > I will hack something this weekend to see if I can bring up a kafka > scheduler on mesos using dockerized brokers. . > > > > -- > κρισhναν > > On Thu, Jul 7, 2016 at 7:29 PM, Christian Posta > > wrote: > > > One thing I can think of is Kafka likes lots of OS page cache. > Dockerizing > > from the standpoint of packaging configs is a good idea, just make sure > if > > you're running many brokers together on the same host, they've got enough > > resources (CPU/Mem) so they don't starve each other. > > > > On Thu, Jul 7, 2016 at 2:30 AM, Krish wrote: > > > >> Hi, > >> I am currently testing a custom docker volume driver plugin for AWS > >> EFS/EBS > >> access and mounting. So, running kafka broker inside a container makes > >> will > >> ease up a lot of configuration issues wrt storage for me. > >> > >> Are there any pros and cons of dockerizing kafka broker? > >> Off the top of my head, since kafka forms the base of our setup, I can > >> think of making is use the host networking stack, and increase ulimits > for > >> the container. > >> I would like to know if and when kafka becomes greedy and cannibalizes > >> resources; I can also ensure that it runs on a dedicated machine. > >> > >> Thanks. > >> > >> Best, > >> Krish > >> > > > > > > > > -- > > *Christian Posta* > > twitter: @christianposta > > http://www.christianposta.com/blog > > http://fabric8.io > > > > >
Re: Pros and cons of dockerizing kafka brokers?
Thanks, Christian. I am currently reading about kafka-on-mesos. I will hack something this weekend to see if I can bring up a kafka scheduler on mesos using dockerized brokers. . -- κρισhναν On Thu, Jul 7, 2016 at 7:29 PM, Christian Posta wrote: > One thing I can think of is Kafka likes lots of OS page cache. Dockerizing > from the standpoint of packaging configs is a good idea, just make sure if > you're running many brokers together on the same host, they've got enough > resources (CPU/Mem) so they don't starve each other. > > On Thu, Jul 7, 2016 at 2:30 AM, Krish wrote: > >> Hi, >> I am currently testing a custom docker volume driver plugin for AWS >> EFS/EBS >> access and mounting. So, running kafka broker inside a container makes >> will >> ease up a lot of configuration issues wrt storage for me. >> >> Are there any pros and cons of dockerizing kafka broker? >> Off the top of my head, since kafka forms the base of our setup, I can >> think of making is use the host networking stack, and increase ulimits for >> the container. >> I would like to know if and when kafka becomes greedy and cannibalizes >> resources; I can also ensure that it runs on a dedicated machine. >> >> Thanks. >> >> Best, >> Krish >> > > > > -- > *Christian Posta* > twitter: @christianposta > http://www.christianposta.com/blog > http://fabric8.io > >
Re: Pros and cons of dockerizing kafka brokers?
One thing I can think of is Kafka likes lots of OS page cache. Dockerizing from the standpoint of packaging configs is a good idea, just make sure if you're running many brokers together on the same host, they've got enough resources (CPU/Mem) so they don't starve each other. On Thu, Jul 7, 2016 at 2:30 AM, Krish wrote: > Hi, > I am currently testing a custom docker volume driver plugin for AWS EFS/EBS > access and mounting. So, running kafka broker inside a container makes will > ease up a lot of configuration issues wrt storage for me. > > Are there any pros and cons of dockerizing kafka broker? > Off the top of my head, since kafka forms the base of our setup, I can > think of making is use the host networking stack, and increase ulimits for > the container. > I would like to know if and when kafka becomes greedy and cannibalizes > resources; I can also ensure that it runs on a dedicated machine. > > Thanks. > > Best, > Krish > -- *Christian Posta* twitter: @christianposta http://www.christianposta.com/blog http://fabric8.io
Pros and cons of dockerizing kafka brokers?
Hi, I am currently testing a custom docker volume driver plugin for AWS EFS/EBS access and mounting. So, running kafka broker inside a container makes will ease up a lot of configuration issues wrt storage for me. Are there any pros and cons of dockerizing kafka broker? Off the top of my head, since kafka forms the base of our setup, I can think of making is use the host networking stack, and increase ulimits for the container. I would like to know if and when kafka becomes greedy and cannibalizes resources; I can also ensure that it runs on a dedicated machine. Thanks. Best, Krish
