First, find a use that makes sense to you. Are you a developer and want to
focus on building applications and tools that interact with and through kafka?
Are you a platform person, who wants to figure out how to build a data ingest
platform and connect various systems and data types through
So long as you put some basic monitoring in place, it should run nicely with
very little intervention and let you be confident everything is as it should be.
Key things to watch:
* disk space - a disk filling up really makes things difficult for you. Make
sure your retention fits your
I want to say that the metrics only show up when the first message comes in,
but I could be thinking of another tool.
Try sending a message to the broker and see if metrics appear?
t.
-Original Message-
From: Wollert, Fabian [mailto:fabian.woll...@zalando.de]
Sent: Thursday, December
The quota page is here: http://kafka.apache.org/documentation.html#design_quotas
"By default, each unique client-id receives a fixed quota in bytes/sec as
configured by the cluster (quota.producer.default, quota.consumer.default)"
I also noticed there's been a change in the replication
Hi Debraj,
A couple things you could try.
Given your design
Another alternative is to checkout Kaboom
https://github.com/blackberry/KaBoom
It uses a pared down kafka consumer library to pull data from Kafka and write
it to defined (and somewhat dynamic) hdfs paths in a custom (and changeable)
avro schema we call boom. It uses kerberos for
Hi Andrew,
Are you using Snappy Compression by chance? When we tested the 0.8.2.1 upgrade
initially we saw similar results and tracked it down to a problem with Snappy
version 1.1.1.6 (https://issues.apache.org/jira/browse/KAFKA-2189). We’re
running with Snappy 1.1.1.7 now and the
Flume is one of the options to do this, yes. There are a (growing) number of
tools you could use to get logs in to Kafka, depending on your environment,
needs, level of skill, deployment size, applications, log volume:
(in no particular order)
Kafkacat :
You can also checkout Klogger (https://github.com/blackberry/Klogger), which
will take input from a TCP port or a file.
Todd.
-Original Message-
From: Jason Gustafson [mailto:ja...@confluent.io]
Sent: Monday, July 13, 2015 20:09
To: users@kafka.apache.org
Subject: Re: Kafka producer
producer/consumer
library which is used by both Klogger and KaBoom.
The project is still active internally, with frequent releases.
Please have a look if you're interested. We welcome any feedback or
contributions.
Cheers,
Todd
Todd Snyder
Senior Infrastructure Specialist, Big Data Platform
10 matches
Mail list logo
