Re: Kafka on AWs EMR

[Andrew Ehrlich]

EMR is a product optimized for elastic scaling of clusters - using amazon's
pre-defined AMIs. Hadoop and Spark work well on this, but not so much
Kafka, which isn't so easy to automatically add/remove nodes from. The best
bet is still to use EC2 with your own Kafka AMIs or predefined third party
ones. Take a look at the AWS Cloudformation product for a handy way to
manage a cluster of EC2 images.

On Sun, Jul 10, 2016 at 7:49 PM, Snehalata Nagaje <
snehalata.nag...@harbingergroup.com> wrote:

>
>
>
> Though it has nothing to do map reduce, I was referingto EMR machines vs
> EC2 instance on amaozone?
>
> We an install other application on aws emr apart from what AWS provides,
> isn't it right?
>
> - Original Message -
> From: r...@chartbeat.com
> To: users@kafka.apache.org
> Sent: Sunday, July 10, 2016 9:25:30 PM
> Subject: Re: Kafka on AWs EMR
>
> Kafka is in no way related to map reduce, this doesn't really make any
> sense. You could have consumers running in Hadoop listening to Kafka topics
> - see the Samza project.
>
> > On Jul 10, 2016, at 11:16 AM, Snehalata Nagaje <
> snehalata.nag...@harbingergroup.com> wrote:
> >
> >
> >
> > Hi All,
> >
> >
> > I wanted to setup kafka on AWS EMR?
> >
> > But I could not find any documentation. Can you please help me on this?
> >
> > Thanks,
> > Snehalata
>

Re: kafka-web-console goes down regularly

[Andrew Ehrlich]

I saw performance issues with the web console whenever using a large 
number of partitions (1000 partitions).


On 2/3/15 12:09 PM, Sa Li wrote:

Hi, All

I am currently using kafka-web-console to monitor the kafka system, it get
down regularly, so I have to restart it every few hours which is kinda
annoying. I downloaded two versions

https://github.com/claudemamo/kafka-web-console
http://mungeol-heo.blogspot.ca/2014/12/kafka-web-console.html

And I started by

play start (using 9000), or play "start -Dhttp.port=8080", either version
can make it work fine at the beginning, but down after few hours. I am
thinking use upstart to make it up automatically, but it should not be what
is suppose to be, any idea to fix the problem, or I did something wrong?

thanks

Re: latency - how to reduce?

[Andrew Ehrlich]

What instance types are you using in EC2? Are the drives EBS?

On 1/5/15 11:06 PM, Shlomi Hazan wrote:

Will do. What did you have in mind? just write a big file to disk and
measure the time it took to write? maybe also read back? using specific
API's?
Apart from the local Win machine case, are you aware of any issues with
Amazon EC2 instances that may be causing that same latency in production?
Thanks,
Shlomi

On Tue, Jan 6, 2015 at 4:04 AM, Jun Rao  wrote:


Not setting "log.flush.interval.messages" is good since the default gives
the best latency. Could you do some basic I/O testing on the local FS in
your windows machine to make sure the I/O latency is ok?

Thanks,

Jun

On Thu, Jan 1, 2015 at 1:40 AM, Shlomi Hazan  wrote:


Happy new year!
I did not set "log.flush.interval.messages".
I also could not find a default value in the docs.
Could you explain about that?
Thanks,
Shlomi

On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao  wrote:


What's your setting of log.flush.interval.messages on the broker?

Thanks,

Jun

On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan 

wrote:

Hi,
I am using 0.8.1.1, and I have hundreds of msec latency at best and

even

seconds at worst.
I have this latency both on production, (with peak load of 30K

msg/sec,

replication = 2 across 5 brokers, acks = 1),
and on the local windows machine using just one process for each of
producer, zookeeper, kafka, consumer.
Also tried batch.num.messages=1 and producer.type=sync on the local

machine

but saw no improvement.
How can I push latency down to several millis, at least when running

local?

Thanks,
Shlomi

Re: How to use kafka as flume source.

[Andrew Ehrlich]

The author may or may not have upgraded the project to be compatible with
0.8.1. You could message the author directly to ask.


On Sun, Aug 3, 2014 at 6:53 AM, rafeeq s  wrote:

> Thanks.
> But above link example uses  flume-ng  to
> communicate with *kafka 0.7,2  >*.
>
> I am using *kafka 0.8.1*. Please advice an example of flume-ng with kafka
> 0.8.1.
>
> Thanks in advance
>
> Regards,
>
> Rafeeq S
> *(“What you do is what matters, not what you think or say or plan.” )*
>
>
>
> On Sun, Aug 3, 2014 at 4:40 PM, Sharninder  wrote:
>
> > https://github.com/baniuyao/flume-ng-kafka-source
> >
> >
> >
> > On Sun, Aug 3, 2014 at 6:15 PM, rafeeq s  wrote:
> >
> > > Hi,
> > >
> > > We are planning to use kafka as *flume source*. Please advice me, how
> to
> > > use kafka as source in flume.
> > >
> > > please share if there is any best example of *flume- kafka source- hdfs
> > > sink*.
> > >
> > > Regards,
> > >
> > > Rafeeq S
> > > *("What you do is what matters, not what you think or say or plan." )*
> > >
> >
>