z Intel Core i7
>
>
> Thanks,
> Neha
>
>
>
> On Tue, Jun 4, 2013 at 5:56 AM, S Ahmed <sahmed1...@gmail.com> wrote:
>
> > I have a 1st gen i7 wit 8GB ram, and mine takes 77 seconds:
> >
> > >clean
> > >++2.9.2 package
> >
> >
The book says Feb. 2016 release date, looks like the producer is slower
than the consumers (sorry couldn't resist)
On Tue, Mar 29, 2016 at 11:56 AM, S Ahmed <sahmed1...@gmail.com> wrote:
> It was some pakt pub one.
>
> Yeah I am waiting for that book to be released!
>
>
o> is an worthy
> investment.
>
> B
> > On 29 Mar 2016, at 03:40, S Ahmed <sahmed1...@gmail.com> wrote:
> >
> > Hello,
> >
> > This may be a silly question for some but here goes :)
> >
> > Without real production experience, what steps do yo
Hello,
This may be a silly question for some but here goes :)
Without real production experience, what steps do you suggest one take to
really have some solid skillz in kafka?
I tend to learn in a structured way, but it just seems that since kafka is
a general purpose tool there isn't really a
Hello,
I have read a few sites that *might* be using kafka for a real-time chat
system.
I say might because in their job descriptions they hinted toward this.
If there are people out there using kafka as a backend for a real-time chat
system, can you explain at a high level how the data flows?
Hi,
I have a light load scenerio but I am starting off with kafka because I
like how the messages are durable etc.
If I have 4-5 topics, am I required to create the same # of consumers? I
am assuming each consumer runs in a long-running jvm process correct?
Are there any consumer examples
dependency?
Thanks,
Jun
On Thu, Oct 9, 2014 at 8:20 AM, S Ahmed sahmed1...@gmail.com wrote:
Hi,
I was wondering if the zookeeper library (zkutils.scala etc) was designed
in a more modular way, would it make it possible to run a more lean
version of kafka?
The idea is I want to run
Hi,
A few questions on timing related issues when certain parts of kafka go
down.
1. If zookeeper goes down, then I bring it back online, do I have to
restart the brokers?
2. If the brokers go down, producers will be erroring out. When the
brokers are back online, do I have to restart the
My app is in scala and a quick search on serializing a scala class seems to
have potential issues with different versions of scala (I could be wrong as
I did a quick search).
Is it generally just a better idea to use plain old java classes for kafka
messages?
i.e. I simply use jackson like:
I'd love to get some insights on how things work at linkedin in terms of
your web servers and kafka producers.
You guys probably connect to multiple kafka clusters, so let's assume you
are only connecting to a single cluster.
1. do you use a single producer for all message types/topics?
2. For
I found this embedded kafka example online (
https://gist.github.com/mardambey/2650743) which I am re-writing to work
with 0.8
Can someone help me re-write this portion:
val cons = new SimpleConsumer(localhost, 9090, 100, 1024)
var offset = 0L
var i = 0
while (true) {
val
)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
On Fri, Jun 13, 2014 at 4:51 PM, S Ahmed sahmed1...@gmail.com wrote:
I found
Is there a simple example (scala preferred) where there is a consumer that
is written to run as a deamon i.e. it keeps as open connection into a
broker and reads off new messages
On Thu, Jun 12, 2014 at 11:13 AM, S Ahmed sahmed1...@gmail.com wrote:
Is there a simple example (scala preferred) where there is a consumer
that
is written to run as a deamon i.e. it keeps as open connection into a
broker and reads off new messages
--
-- Guozhang
For those of you hosting on ec2, could someone suggest a minimum
recommended setup for kafka? i.e. the # and type of instance size that you
would say is the bare minimum to get started with kafka in ec2.
My guess is the suggest route is the m3 instance type?
How about:
m3.medium 1 cpu, 3.75GB
I swear I read that Jay Kreps wrote somewhere that consumers now write
their offsets in a logfile (not in zookeeper).
Is this true or did I misread? Sorry I can't find the article I was
reading.
:
It is not recommended to install both kafka and zookeeper on the same box
as both would fight for the available memory and performance will degrade.
Thanks
Neha
On Mon, May 19, 2014 at 7:29 AM, S Ahmed sahmed1...@gmail.com wrote:
Hi,
I like how kafka operates, but I'm wondering
Hi,
I like how kafka operates, but I'm wondering if it is possible to run
everything on a single ec2 instance with 7.5 GB RAM.
So that would be zookeeper and a single kafka broker.
I would have a separate server to consume from the broker.
Producers would be from my web servers.
I don't want
How about the following use case:
Just before the producer actually sends the payload to kakfa, could an
event be exposed that would allow one to loop through the messages and
potentially delete some of them?
Example:
Say you have 100 messages, but before you send these messages to kakfa, you
Sorry I'm not a ops person, but what tools do you use to monitor traffic
between servers?
On Tue, Feb 4, 2014 at 11:46 PM, Carl Lerche m...@carllerche.com wrote:
Hello,
I'm running a 0.8.0 Kafka cluster of 3 servers. The service that it is
for is not in full production yet, so the data
I'm trying to trace through the codebase and figure out where exactly the
block occurs in the high level consumer?
public void run() {
ConsumerIteratorbyte[], byte[] it = m_stream.iterator();
while (it.hasNext())
System.out.println(Thread + m_threadNumber + : + new
I have briefly looked at storm, but just a quick question, storm seems to
have all these workers but they way it seems to me the order in which these
items are processed off the queue is very random correct?
In my use case order is very important so using something like storm would
not be
as a clustered, consistent, highly available
store for this sort of data and it works extremely well. Redis wasn't and I
don't know anyone using Redis in production, including me, who doesn't have
stories of Redis losing data. I'm sticking with ZK.
On Tue, Dec 17, 2013 at 10:57 AM, S Ahmed sahmed1
, or simply a restart, it doesn't matter.
Philip
On Mon, Dec 9, 2013 at 12:28 PM, S Ahmed sahmed1...@gmail.com wrote:
I was hoping people could comment on how they handle the following
scenerios:
1. Storing the last successfully processed messageId/Offset. Are people
using mysql
Is there a book or this was just an idea?
On Mon, Mar 25, 2013 at 12:42 PM, Chris Curtin curtin.ch...@gmail.comwrote:
Thanks Jun,
I've updated the example with this information.
I've also removed some of the unnecessary newlines.
Thanks,
Chris
On Mon, Mar 25, 2013 at 12:04 PM, Jun
Arthur mum...@gmail.com wrote:
There was some talk a few months ago, not sure what the current status
is.
On 12/10/13 10:01 AM, S Ahmed wrote:
Is there a book or this was just an idea?
On Mon, Mar 25, 2013 at 12:42 PM, Chris Curtin curtin.ch...@gmail.com
wrote:
Thanks Jun
, as is standard with Kafka.
Our systems are idempotent, so we only store offsets when the message is
fully processed. If this means we occasionally replay a message due to some
corner-case, or simply a restart, it doesn't matter.
Philip
On Mon, Dec 9, 2013 at 12:28 PM, S Ahmed sahmed1...@gmail.com
I remember a while back Jay was looking for someone to work on producer (or
was it consumer) statisitics which was going to use metrics.
Was this ever implemented?
Is anyone out there running a single broker kafka setup?
How about with only 8 GB RAM?
I'm looking at one of the better dedicated server prodivers, and a 8GB
server is pretty much what I want to spend at the moment, would it make
sense going this route?
This same server would also potentially be
can be 10s of KB. This is mostly because we
batch a set of messages and send them as a single compressed message.
Thanks,
Jun
On Mon, Oct 7, 2013 at 7:44 AM, S Ahmed sahmed1...@gmail.com wrote:
When people using message queues, the message size is usually pretty
small.
I want to know
I have a use case where thousands of servers send status type messages,
which I am currently handling real-time w/o any kind of queueing system.
So currently when I receive a message, and perform a md5 hash of the
message, perform a lookup in my database to see if this is a duplicate, if
not, I
Similar, yet different. I like it!
On Mon, Jul 22, 2013 at 1:25 PM, Jay Kreps jay.kr...@gmail.com wrote:
Yeah, good point. I hadn't seen that before.
-Jay
On Mon, Jul 22, 2013 at 10:20 AM, Radek Gruchalski
radek.gruchal...@portico.io wrote:
296 looks familiar:
When you guys refactored the offset to be human friendly i.e. numerical
versus the byte offset, what was involved in that refactor? Is there a
wiki page for that?
I'm guessing there was a index file that was created for this, or is this
currently managed in zookeeper?
This wiki is related to
Was this recorded by any chance?
On Wed, Jun 26, 2013 at 11:22 AM, Jun Rao jun...@gmail.com wrote:
Hi, Everyone,
We have finalized our agenda of the meetup Thursday evening, with Speakers
from LinkedIn, RichRelevance, Netflix and Square. Please RSVP to the meetup
link below. For remote
Hi,
In my application, I am storing user events, and I want to partition the
storage by day.
So at the end of a day, I want to take that file and ship it to s3 or
another server as a backup. This way I can replay the events for a
specific day if needed.
These events also have to be in order.
I understand that there are no guarantees per say that a message may be a
duplicate (its the consumer's job to guarantee that), but when it comes to
message order, is kafka built in such a way that it is impossible to get
messages in the wrong order?
Certain use cases might not be sensitive to
Is the code you used to benchmark open source by any chance?
On Tue, May 28, 2013 at 4:29 PM, Jason Weiss jason_we...@rapid7.com wrote:
Nope, sorry.
From: S Ahmed [sahmed1...@gmail.com]
Sent: Tuesday, May 28, 2013 15:47
To: users@kafka.apache.org
Interesting topic.
How would buffering in RAM help in reality though (just trying to work
through the scenerio in my head):
producer tries to connect to a broker, it fails, so it appends the message
to a in-memory store. If the broker is down for say 20 minutes and then
comes back online, won't
I thought since the offsets in .8 are numeric and not byte offsets like in
0.7x, you can simply just take say the current offset - 1.
On Tue, Mar 19, 2013 at 12:16 PM, Neha Narkhede neha.narkh...@gmail.comwrote:
Jim,
You can leverage the ExportZkOffsets/ImportZkOffsets tools to do this.
I guess the challenge would be that kafka is still in version 0.8, so by
the time your book comes out they might be at version 1.0 i.e. its a moving
target
Sounds like a great idea though!
On Tue, Mar 19, 2013 at 12:20 PM, Jun Rao jun...@gmail.com wrote:
Hi, David,
At LinkedIn, committers
Ok so it might be an issue somewhere in the pipeline (I'm guessing memory
issues?).
They are xml files, and that 30-100 was uncompressed.
On Tue, Jan 29, 2013 at 12:28 PM, Neha Narkhede neha.narkh...@gmail.comwrote:
At linkedin, what is the largest payload size per message you guys have
in
should monitor ZK
request latency and GCs.
Thanks,
Jun
On Fri, Dec 28, 2012 at 7:27 AM, S Ahmed sahmed1...@gmail.com wrote:
Curious what kind of uptime have you guys experienced using kafka?
What sort of monitoring do you suggest should be in place for kafka?
If the service crashes
help anyone? :)
Much much appreciated!
On Tue, Dec 11, 2012 at 12:03 AM, S Ahmed sahmed1...@gmail.com wrote:
BTW, where exactly will the broker be writing these messages? Is it in a
/tmp folder?
On Tue, Dec 11, 2012 at 12:02 AM, S Ahmed sahmed1...@gmail.com wrote:
Neha,
But what do I
,
Neha
On Mon, Dec 10, 2012 at 7:31 PM, S Ahmed sahmed1...@gmail.com wrote:
Hi,
So I followed the instructions from here:
https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup
So I pulled down the latest from github, ran;
sbt
update
idea
open it up in idea
BTW, where exactly will the broker be writing these messages? Is it in a
/tmp folder?
On Tue, Dec 11, 2012 at 12:02 AM, S Ahmed sahmed1...@gmail.com wrote:
Neha,
But what do I need to start before running the tests, I tried to run the
test testAsyncSendCanCorrectlyFailWithTimeout but I got
45 matches
Mail list logo
