Which version of ZK are you using?
Thanks,
Jun
On Fri, Jun 7, 2013 at 12:20 PM, Evan Chan wrote:
> [ Sorry if this mail is duplicated, this is my fourth try sending this
> message]
>
> Hey guys,
>
> I sincerely apologize if this has been covered before, I haven't quite
> found a similar situa
*We have a mixture of:*
ERROR: something bad happened
*Some logs are "actions":*
user upload a file
user collaborated a file
*Some logs are metrics:*
counter:webapp.rps:+1
-Jonathan
On Fri, Jun 7, 2013 at 8:12 PM, Mark wrote:
> Are these always log files in the sense of log files or do th
Are these always log files in the sense of log files or do they also contain
some event data.. i.e. Product A was purchased or User A just signed in, etc?
On Jun 7, 2013, at 6:53 PM, Jonathan Creasy wrote:
> Correct, we essentially use the logs as an additional buffer in case of
> outage in th
Correct, we essentially use the logs as an additional buffer in case of
outage in the pipeline. Typically though, messages are produces as soon as
they are written.
-Jonathan
On Fri, Jun 7, 2013 at 6:06 PM, Mark wrote:
> Ok so in your use case instead of your application(s) writing directly t
Thanks so much for your replies. This has been a great help understanding
Rabbit better with having very little experience with it. I have a few
follow up comments below.
> While you are correct the payload is a much bigger concern, managing the
> metadata and acks centrally on the broker acros
Ok so in your use case instead of your application(s) writing directly to Kafka
you instead have a separate process running that will tail log files and ship
them over to Kafka. Is that correct?
On Jun 7, 2013, at 5:33 PM, Jonathan Creasy wrote:
> I recommend Kafka or Flume-NG for this.
>
> O
I recommend Kafka or Flume-NG for this.
Our Analytics team is using a Kafka Producer on each server to tail logs
and ship them to Kafka. We use Oozie to schedule a MapReduce consumer every
few minutes to read all the Kafka topics into HDFS.
We use Kafka as a buffer, we keep a few weeks of data th
Like I said, Im a bit confused. I see the terms "events", "messages" and "logs"
and not quite sure what to make of it.
We are trying to determine the best way to aggregate all of our logs for
processing in Hadoop. Kafka seems to fit this bill nicely however I want to
know If its suited for othe
On Sat, Jun 8, 2013 at 1:08 AM, Mark wrote:
> Im a bit confused on the concept of a "message" in Kafka. How does this
> differ, if at all, from a message in RabbitMQ? It seems to me that Kafka is
> better suited for very write intensive "messages" like log data but RabbitMQ
> may be a better f
Im a bit confused on the concept of a "message" in Kafka. How does this
differ, if at all, from a message in RabbitMQ? It seems to me that Kafka is
better suited for very write intensive "messages" like log data but RabbitMQ
may be a better fit for traditional "messages"… i.e. "Product Purchase
Jonathan,
On Fri, Jun 7, 2013 at 7:42 PM, Jonathan Hodges wrote:
> Sorry I forgot to add this RabbitMQ link as well as it also seems to
> indicate the copying of messages to multiple queues with a topic exchange.
>
> http://www.rabbitmq.com/tutorials/tutorial-five-python.html
"Copy" is not menti
Jonathan,
On Fri, Jun 7, 2013 at 7:03 PM, Jonathan Hodges wrote:
> Hi Alexis,
>
> I appreciate your reply and clarifications to my misconception about
> Rabbit, particularly on the copying of the message payloads per consumer.
Thank-you!
> It sounds like it only copies metadata like the cons
IIRC, I think I tried to use stunnel with Kafka once and it worked fine and the
configuration wasn't too bad, at least for a simple configuration.
-Original Message-
From: Dragos Manolescu [mailto:dragos.manole...@servicenow.com]
Sent: Friday, June 07, 2013 4:51 PM
To: users@kafka.apache
Thank you Marc (and others) for jumping in and sharing your perspectives!
A feature that Kafka doesn't currently support while RabbitMQ does (since
about a year ago, I don't remember exactly) is SSL support. I realize that
one can set up a tunnel between data centers, etc.; that would require
more
Which version of ZK are you using ?
On Fri, Jun 7, 2013 at 12:15 PM, Evan Chan wrote:
> Hey guys,
>
> I sincerely apologize if this has been covered before, I haven't quite
> found a similar situation.
>
> We are using Kafka 0.7.2 in production, and we are using the ZK high level
> Scala consum
[ Sorry if this mail is duplicated, this is my fourth try sending this
message]
Hey guys,
I sincerely apologize if this has been covered before, I haven't quite
found a similar situation.
We are using Kafka 0.7.2 in production, and we are using the ZK high level
Scala consumer. However, we fin
Hey guys,
I sincerely apologize if this has been covered before, I haven't quite
found a similar situation.
We are using Kafka 0.7.2 in production, and we are using the ZK high level
Scala consumer. However, we find the ZK consumer very unstable. It would
work for one or two weeks, then sudden
Sorry I forgot to add this RabbitMQ link as well as it also seems to
indicate the copying of messages to multiple queues with a topic exchange.
http://www.rabbitmq.com/tutorials/tutorial-five-python.html
Maybe I am misunderstanding and topic exchange wouldn't be the approach to
take with Rabbit i
Hi Alexis,
I appreciate your reply and clarifications to my misconception about
Rabbit, particularly on the copying of the message payloads per consumer.
It sounds like it only copies metadata like the consumer state i.e.
position in the topic messages. I don’t have experience with Rabbit and
wa
Jun,
On Fri, Jun 7, 2013 at 4:23 PM, Jun Rao wrote:
> Hi, Alex,
>
> Thanks for sharing your thoughts here. Your understand of Kafka is correct
> and your analysis is very helpful. Just a couple of followup questions on
> RabbitMQ.
>
> 1. How do people typically scale out RabbitMQ? In Kafka, we ha
Hi, Alex,
Thanks for sharing your thoughts here. Your understand of Kafka is correct
and your analysis is very helpful. Just a couple of followup questions on
RabbitMQ.
1. How do people typically scale out RabbitMQ? In Kafka, we have this
notion of a cluster that can include multiple brokers. Bro
Hi,
looking at the stack trace It seems to me you are using Kafka 0.7.0.
Well, I think the issue has something to do with broker(s) registration
(i.e Zookeeper) so you should check the startup messages on Kafka and
eventually the Zookeeper logs, I'm quite sure you will find something there.
Th
Hi
Alexis from Rabbit here. I hope I am not intruding!
It would be super helpful if people with questions, observations or
moans posted them to the rabbitmq list too :-)
A few comments:
* Along with ZeroMQ, I consider Kafka to be one of the interesting and
useful messaging projects out there.
Nice of you to reply Alexis and clarify things! FWIW, I personally like
Rabbit very much and I am pushing to use it for other purposes at my
company. It's flexibility, ease of use and even documentation is really top
if you compare with other options.
I might have explained some of my points a bit
Hi
Alexis from Rabbit here. I hope I am not intruding!
It would be super helpful if people with questions, observations or
moans posted them to the rabbitmq list too :-)
A few comments:
* Along with ZeroMQ, I consider Kafka to be one of the interesting and
useful messaging projects out there.
25 matches
Mail list logo
