Yeah the issue was mostly that literally ALL of the docs use / as the
chroot. People kept getting tripped up with the ZK URL with /kafka, it's a
bit funky eg:
host1.domain.com:2181,host2.domain.com:2181,host3.domain.com:2181/kafka. So
we opted to change it back to what everyone else is doing by
Yes, that's the issue. Currently, topics can be auto-created on
TopicMetadataRequest, which can be issued from both the producer and the
consumer. To prevent that, you would need to stop the producer and the
consumer before deleting a topic. We plan to address this issue once we
have a separate
There could be another case where if you have auto.create.topics.enable
to set to true ( its true by default) . Any TopicMetadataRequest can
recreate topics. So if you issued a delete topic command and you have
producers running or consumers? too which is issuing a
TopicMetadataRequest than the
Jun,
I made an attempt at fixing that issue as part of this JIRA
https://issues.apache.org/jira/browse/KAFKA-1507 .
As Jay pointed out there should be admin api if there is more info on
this api I am interested in adding/fixing this issue.
Thanks,
Harsha
On Mon, Jan 26, 2015, at
shouldn't the new consumer api be removed from the 0.8.2 code base then?
On Fri, Jan 23, 2015 at 10:30 AM, Joe Stein joe.st...@stealth.ly wrote:
The new consumer is scheduled for 0.9.0.
Currently Kafka release candidate 2 for 0.8.2.0 is being voted on.
There is an in progress patch to the
we are pulling from a DB, maybe 100k rows at a time. for each row we send
it to kafka.
now the problem is , if any errors happen during the db pulling (such as an
earthquake), we stop.
next time we wake up. we can't write the same record again into kafka,
otherwise there might be over counting.
hi All,
I am facing the same issue.. i have got Kafka_2.9.2-0.8.1.1.jar deployed
in my application.
My Operation team had reported significant increase in the directory where
Kafak-log folder exists . when i checked the size of parent directory is it
99GB (100%) utilized.. This was very
Hey Jason,
Is it an option for you to do the following:
- Bounce in a config change to the brokers to turn off auto-create
- (Batch)-delete the topic(s)
- Wait long enough for consumers to rebalance (after which they will
no longer consume the topic(s))
- Bounce in a config change to the
So, this is rather disappointing, especially since topic deletion is really
the primary feature in 0.8.2 I'm interested in. The topic I was trying to
delete above had no data for many months. The consumer which is triggering
recreation of that topic has been restarted several times since that
The new consumer api is actually excluded from the javadoc that we generate.
Thanks,
Jun
On Mon, Jan 26, 2015 at 11:54 AM, Jason Rosenberg j...@squareup.com wrote:
shouldn't the new consumer api be removed from the 0.8.2 code base then?
On Fri, Jan 23, 2015 at 10:30 AM, Joe Stein
Hi all,
I am not sure if we have discussed about this before, but recently I
realized that we have introduced boot dependency of the kafka-server
specified by the bootstrap.servers config in the new producer. More
specifically, although in the old producer we also have a similar config
for
Matve wr should add to the documentation experimental so folks that don't
know understand.
/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop
Sa,
I believe your questions have mostly been answered by Ewen, and sorry for
getting late on this.
As you notice the current system test's out-of-the-box experience is not
very good, and we are proposing ways to improve that situation:
KAFKA-1748
It then seems to me that delete-topic will not actually work smoothly
until create topic request is added since it is too much to require people
to turn off their clients while deleting topics. In this case shall we make
it clear in the release docs of in 0.8.2, or even still mark it as
Hi, Jason,
I am not sure that we should delay the 0.8.2 release. The reasons are (1)
There are other features such as the new java producer and Kafka-based
offset management that are potentially useful to people. (2) It may take
some time to completely fix the issue with deleting topic since it
Hi Kafka Team,
I just wanted to bring this to your attention regarding Java New Producer
limitation compare to old producer.
a) Partition Increasing is limited to configured memory allocation.
buffer.memory
batch.size
The maximum partition you could have before impacting (New Java Producers)
Hi Neha,
I sent an e-mail earlier today, but noticed now that it didn't actually go
through.
Anyhow, I've attached two files, one with output from a 10 minute run and
one with output from a 30 minute run. Realized that maybe I should've done
one or two runs with 0.8.1.1 as well, but
unsubscribe
On Mon, Jan 26, 2015 at 8:14 AM, Harsha ka...@harsha.io wrote:
Jun,
I made an attempt at fixing that issue as part of this JIRA
https://issues.apache.org/jira/browse/KAFKA-1507 .
As Jay pointed out there should be admin api if there is more info on
this api I am
Hello Everyone,
I recently watched this video by Tim Kellogg:
https://www.youtube.com/watch?v=VoTclkxSago.
1) Has anyone integrated MQTT and Kafka? If so how did the architecture
look like? I am confused as to how to get messages from the MQTT broker to
the Kafka broker.
2) Has anyone used this
+1 (binding)
artifacts and quick start look good. I ran in some client code, minor edits
from 0-8.2-beta https://github.com/stealthly/scala-kafka/pull/26
On Mon, Jan 26, 2015 at 3:38 AM, Manikumar Reddy ku...@nmsworks.co.in
wrote:
+1 (Non-binding)
Verified source package, unit tests, release
Actually, I think the deletion of a topic will trigger a rebalance on all
wildcard subscribers. The time to complete the rebalance depends on the #
of topic/partitions.
Thanks,
Jun
On Mon, Jan 26, 2015 at 12:26 PM, Jason Rosenberg j...@squareup.com wrote:
Yeah Joel,
I just thought of that
thanks for the info
yang
On Mon, Jan 26, 2015 at 2:09 PM, Jay Kreps jay.kr...@gmail.com wrote:
We did a relatively complete prototype but it isn't integrated into the
mainline code yet and there isn't a target release date. There is rather a
lot of testing and compatability work that would
Thank you for reply Guozhang, right now I can get it work out of box, run
testcase_1 on VM and access production. However, from my point of view, we
really like to test the existing configs on productions, which means for
example, replica_basic_test.py won't start zookeeperkafka since I want to
Oh, yes, I guess I thought you meant that construction of the client would
block on the metadata request.
I don't personally think that is a problem because if it fails it will
retry in the background, right?
But actually I think this is probably violating another desirable criteria
we had
I was planning to start a new thread by my experience is similar to the one
described here. I am currently using 0.8.2-beta and my setup has a producer
producing to topic DC-DATA, kafka brokers with auto-topic-creation
enabled, and mirrormaker set to replicate any .*-DATA topics. After the
We did a relatively complete prototype but it isn't integrated into the
mainline code yet and there isn't a target release date. There is rather a
lot of testing and compatability work that would have to be done to fully
productionize it. I suspect someone will pick it up in 2015 but I wouldn't
Hey Guozhang,
That line shouldn't cause any connections to Kafka to be established, does
it? All that is doing is creating the Cluster pojo using the supplied
addresses. The use of InetSocketAddress may cause some dns stuff to happen,
though...
-Jay
On Mon, Jan 26, 2015 at 10:50 AM, Guozhang
Joel,
That's probably because console consumer always uses wildcard for
consumption.
Thanks,
Jun
On Mon, Jan 26, 2015 at 1:44 PM, Joel Koshy jjkosh...@gmail.com wrote:
I think this should work even on non-wildcard consumers. (i.e., I just
tried it locally and it appears to work fine).
Hmm.. that's right. completely forgot about that.
On Mon, Jan 26, 2015 at 01:49:33PM -0800, Jun Rao wrote:
Joel,
That's probably because console consumer always uses wildcard for
consumption.
Thanks,
Jun
On Mon, Jan 26, 2015 at 1:44 PM, Joel Koshy jjkosh...@gmail.com wrote:
I
well I guess this is what I need
https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka
anyone knows how far this effort has gone?
thanks
Yang
On Mon, Jan 26, 2015 at 12:06 PM, Yang tedd...@gmail.com wrote:
we are pulling from a DB, maybe 100k rows at a time.
Hi,
Don't use Graphite, so I don't know. Kyle, maybe you can share more info?
What do you mean by reported to Yammer for example? And when you say
Yammer/Graphite, are you trying to say that you are using the Graphite
Reporter? If so, can you try other Yammer Reporters and see if there is a
Thanks a lot Gwen. I bumped up the JVM to 1g on the consumer side and it
works :)
All the consumer belong to the same group and I am using the High level
group API to consume from the kafka. It seems there is some initial meta
data exchange or something about all the partitions are sent to all
Hmm, so you just want to start producer / consumer clients which points to
the existing brokers in production. I think what you described here should
work, but I may be wrong. Please let me know if that does not work.
Guozhang
On Mon, Jan 26, 2015 at 1:50 PM, Sa Li sal...@gmail.com wrote:
If you are using multi-node cluster, then metrics may be reported from
other servers.
pl check all the servers in the cluster.
On Tue, Jan 27, 2015 at 4:12 AM, Kyle Banker kyleban...@gmail.com wrote:
I've been using a custom KafkaMetricsReporter to report Kafka broker
metrics to Graphite. In
While running kafka in production I found an issue where a topic wasn't
getting created even with auto topic enabled. I then went ahead and created
the topic manually (from the command line). I then delete the topic, again
manually. Now my broker won't allow me to either create *the* topic or
I can confirm that the per topic metrics are not coming through to the
yammer metrics registry. I do see them in jmx (via jconsole), but the
MetricsRegistry does not have them.
All the other metrics are coming through that appear in jmx.
This is with single node instance running locally.
Jason
The jmx beans created by Yammer have been changed to the new names.
However, the global mbeans have different names from the topic level
mbeans. The new mbeans have completely different names from the old ones
since they no longer have quotes. Does that answer your question?
Thanks,
Jun
On Mon,
Jay, I don't think this line will bootstrap full metadata (for all topics).
it will just construct the cluster object with bootstrap host. you need to
do metadata.add(topic) to set interest of a topic's partition metadata.
Guozhang, I personally think this is ok. it just do a few DNS lookup or
Kyle,
Also, which reporter are you using for Graphite?
Thanks,
Jun
On Mon, Jan 26, 2015 at 4:08 PM, Kyle Banker kyleban...@gmail.com wrote:
This is still preliminary, but it looks as if the change to metric names
for per-topic metrics (bytes/messages in/out) is preventing these metrics
Hi all, I am currently working on logstash to mongodb.logstash's input
(producer) is kafka and output (consumer) is mongodb.It is worked fine for past
two days.Today it is not working and i got the following error.i googled this
error,still didnt find any correct solution.kindly help me
+1 (Non-binding)
Verified source package, unit tests, release build, topic deletion,
compaction and random testing
On Mon, Jan 26, 2015 at 6:14 AM, Neha Narkhede n...@confluent.io wrote:
+1 (binding)
Verified keys, quick start, unit tests.
On Sat, Jan 24, 2015 at 4:26 PM, Joe Stein
Here is my typical flow:
void run() {
if (simpleConsumer == null) {
simpleConsumer = new SimpleConsumer(host, port, (int) kafkaSocketTimeout,
kafkaRExeiveBufferSize, clientName);
}
try {
// Do stuff with simpleConsumer.
} catch (Exception e) {
if (consumer != null) {
Hi Guozhang,
I am a bit busy at work. When I get the change I'll definitely try to get a
proof of concept going. Not the kafka protocol, but just the buffering and
threading structures, maybe just write to another socket. I think it would
be useful just to get the queueing and buffer management
Yes, we refactored the mbean names in 0.8.2.0 to make them more standard.
Those metrics are now registered under the following names. I did some
quick test and the values do get updated.
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
Meant to write a run loop.
void run() {
while (running) {
if (simpleConsumer == null) {
simpleConsumer = new SimpleConsumer(host, port,
(int) kafkaSocketTimeout, kafkaRExeiveBufferSize, clientName);
}
try {
// Do stuff with simpleConsumer.
} catch (Exception e) {
I've been using a custom KafkaMetricsReporter to report Kafka broker
metrics to Graphite. In v0.8.1.1, Kafka was reporting bytes and messages in
and out for all topics together and for each individual topic.
After upgrading to v0.8.2.0, these metrics are no longer being reported.
I'm only seeing
Thanks, Jun. I'm pretty sure, though not 100% confident, that this has
caused a change in the way that these metrics are reported to Yammer
metrics (I believe that they are stomping on each other). Is that intended
as well?
On Mon, Jan 26, 2015 at 4:13 PM, Jun Rao j...@confluent.io wrote:
Yes,
This is still preliminary, but it looks as if the change to metric names
for per-topic metrics (bytes/messages in/out) is preventing these metrics
from being reported to Yammer/Graphite. If this isn't intentional, it
should probably be addressed before release.
On Wed, Jan 21, 2015 at 9:28 AM,
Hmm, that's not the intention. The per-topic mbeans are definitely
registered by Yammer. So, not sure why it's not reported to Graphite.
Otis, Vladimir,
Do you guys know?
Thanks,
Jun
On Mon, Jan 26, 2015 at 4:08 PM, Kyle Banker kyleban...@gmail.com wrote:
This is still preliminary, but it
It will set the needUpdate flag to true and hence the background Sender
will try to talk to the bootstrap servers.
Guozhang
On Mon, Jan 26, 2015 at 1:12 PM, Jay Kreps jay.kr...@gmail.com wrote:
Hey Guozhang,
That line shouldn't cause any connections to Kafka to be established, does
it? All
50 matches
Mail list logo
