We have make a simple web console to monitor some kafka informations like
consumer offset, logsize.
https://github.com/shunfei/DCMonitor
Hope you like it and offer your help to make it better :)
Regards
Flow
DeleteTopic makes a node in zookeeper to let controller know that there is a
topic up for deletion. This doesn’t immediately delete the topic it can take
time depending if all the partitions of that topic are online and brokers are
available as well. Once all the Log files deleted zookeeper
Hi
What is best practice for adding audit feature in kafka, Is there any
framework available for enabling audit feature at producer and consumer
level and any UI frameworks for monitoring.
tx
SunilKalva
Emmanuel, if it helps, here's a little more detail on the hardware spec we
are using at the moment:
12 CPU (HT enabled)
64 GB RAM
16 x 1TB SAS drives (2 are used as a RAID-1 set for the OS, 14 are a
RAID-10 set just for the Kafka log segments)
We don't colocate any other applications with Kafka
With MM, the source and destination cluster can choose different number of
partitions for the mirrored topic, and hence messages may be re-grouped in
the destination cluster. In addition, let's say you have two MMs piping
data to the same destination from two sources, the ordering of which
Thanks for your prompt response.
In my check for topicExists i will add a check for topic in
DeleteTopicsPath.
On Mon, Mar 23, 2015 at 8:21 PM, Harsha ka...@harsha.io wrote:
Just to be clear, one needs to stop producers and consumers that
writing/reading from a topic “test” if they are
Hi All,
Currently we are using kafka_2.8.0-0.8.0-beta1 in our production system. I
am using sync producer with ack=0 to send the events to broker.
but I am seeing most of my producer threads are blocked.
jmsListnerTaskExecutor-818 prio=10 tid=0x7f3f5c05a800 nid=0x1719
waiting for monitor
I created a wiki page that lists all the MySQL replication options that people
posted, plus a couple others. People may/may not find it useful.
https://github.com/wushujames/mysql-cdc-projects/wiki
I wasn't sure where to host it, so I put it up on a Github Wiki.
-James
On Mar 17, 2015, at
Hey Everyone –
Just a reminder about the Meetup tomorrow night @ LinkedIn.
There will be 3 talks:
Offset management - 6:35PM Joel Koshy(LinkedIn)
The Netflix Data Pipeline - 7:05PM - Allen Wang Steven Wu(Netflix)
Best Practices - 7:50PM - Jay Kreps(Confluent)
If you are interested in
We've talked about it a little bit, but part of the problem is that it is
pretty well integrated into our infrastructure, and as such it's hard to
pull it out. I illustrated this a little differently than Jon did in my
latest blog post (http://engineering.linkedin.com/kafka/running-kafka-scale),
I think I worked out the answer to question 1.
java.lang.IllegalMonitorStateException
was thrown due to no ownership of ReentrantLock when trying to call await()
on the lock condition.
Here is the code snippet from the AbstractFetcherThread.scala in trunk
partitionMapLock synchronized {
On deleting a topic via TopicCommand.deleteTopic()
I get Topic test-delete is marked for deletion.
I follow up by checking if the topic exists by using
AdminUtils.topicExists()
which suprisingly returns true.
I expected AdminUtils.TopicExists() to check both BrokerTopicsPath
and
Are there any plans to open source the same? What alternates do we have
here?
We are building an internal auditing framework for our entire big data
pipeline. Kafka is one of the data sources we have (ingested data).
On Mon, Mar 23, 2015 at 1:03 PM, tao xiao xiaotao...@gmail.com wrote:
Linkedin has an excellent tool that monitors lag/data loss/data duplication
and etc. Here is the reference
http://www.slideshare.net/JonBringhurst/kafka-audit-kafka-meetup-january-27th-2015
it is not open sourced though.
On Mon, Mar 23, 2015 at 3:26 PM, sunil kalva kalva.ka...@gmail.com wrote:
Currently we have auto.create.topics.enable set to true by default. If this is
set true any one who is making TopicMetadataRequest can create a topic . As
both producers and consumers can send TopicMetadataRequest which will create a
topic if the above config is true. So while doing deletion if
What happens when producers or consumers are running while the topic
deleting is going on?
On Mon, Mar 23, 2015 at 10:02 AM, Harsha ka...@harsha.io wrote:
DeleteTopic makes a node in zookeeper to let controller know that there is
a topic up for deletion. This doesn’t immediately delete the
Just to be clear, one needs to stop producers and consumers that
writing/reading from a topic “test” if they are trying to delete that specific
topic “test”. Not all producers and clients.
--
Harsha
On March 23, 2015 at 10:13:47 AM, Harsha (harsh...@fastmail.fm) wrote:
Currently we have
Hi Francois,
Looks like this belong storm mailing lists. Can you please send this
question on storm mailing lists.
Thanks,
Harsha
On March 23, 2015 at 11:17:47 AM, François Méthot (fmetho...@gmail.com) wrote:
Hi,
We have a storm topology that uses Kafka to read a topic with 6
If you have feedback, don't hesitate to comment on the JIRA.
On Mon, Mar 23, 2015 at 4:19 PM, Pete Wright pwri...@rubiconproject.com
wrote:
Hi Gwen - thanks for sending this along. I'll patch my local checkout
and take a look at this.
Cheers,
-pete
On 03/20/15 21:16, Gwen Shapira wrote:
Hi Gwen - thanks for sending this along. I'll patch my local checkout
and take a look at this.
Cheers,
-pete
On 03/20/15 21:16, Gwen Shapira wrote:
We have a patch with examples:
https://issues.apache.org/jira/browse/KAFKA-1982
Unfortunately, its not committed yet.
Gwen
On Fri, Mar
Hi Jon,
It the link for the 1/27 meetup you posted works for me, but I haven't
found how to find that same link on the meetup site (there are links that
point to the live stream, which of course is no longer happening!).
Thoughts?
Thanks,
Jason
On Mon, Mar 2, 2015 at 11:31 AM, Jon Bringhurst
21 matches
Mail list logo
