Re: RAID10 for kafka cluster

in my opinion for production I would run with raid10, its true that kafka has durability as to shutdown of brokers, there are exceptions or hiccups, if you want to avoid tons of movement between brokers and connection errors on the clients (which may or not depending on how loaded your cluster is

Re: Scaling up kafka consumers

The kafka fast connector handles this differently than the standard kafka client (which requires one consumer per partition at most), by breaking offsets into consumable ranges which allows one partition to be read by multiple conumers where each consumer uniquely receives a different offset

Re: Kafka SASL_PLAINTEXT and authentication/authorization backend failure

hould give you errors as a general rule. > If you are aware of certain scenarios where it should give an error and it > doesn't, then please file a bug with steps to reproduce. > > Ismael > > On Thu, Jan 19, 2017 at 6:48 PM, Gerrit Jansen van Vuuren < > gerrit...@gmail.com&

Re: Kafka SASL_PLAINTEXT and authentication/authorization backend failure

Hi, I've added kerberos support for https://github.com/gerritjvv/kafka-fast and have seen that the kafka brokers do not send any response if the SASL authentication is not correct or accepted, thus causing the client to hang while waiting for a response from kafka. Some things that might help to

Re: Serious problem

The kafka brokers have a maximum message size limit, this is a protection measure and avoids sending monster messages to kafka. You have two options: 1. On the brokers, increase the max.request.size, default is at ~2mb, making it 5 or even 10 is not an issue normally. Java applications can

Re: Setup Kerberos for Kafka on Ubuntu Linux

et started working on using kerberos without getting senselessly frustrated all the time. On Fri, Dec 30, 2016 at 5:49 PM, Gerrit Jansen van Vuuren < gerrit...@gmail.com> wrote: > make sure kafka1 is the FQN and that the server kafka1 can resolve > properly from you're kerberos serv

Re: Setup Kerberos for Kafka on Ubuntu Linux

make sure kafka1 is the FQN and that the server kafka1 can resolve properly from you're kerberos server, EXAMPLE.COM should be a realm that is configured in krb5.conf and kdc.conf, with the adequate domain mappings for kafka1 to this realm. Kerberos is a pain and there are tons of stuff that can

Re: rebalancing - how to speed it up?

I don't know about speeding up rebalancing, and an hour seems to suggest something is wrong with zookeeper or you're whole setup maybe. if it becomes an unsolvable issue for you, you could try https://github.com/gerritjvv/kafka-fast which uses a different model and doesn't need balancing or

Re: Question regarding dynamic subscriber environment

> connection between broker and subscriber should not be terminated. > Subscriber is free to change his topic interests without closing the > connection. > > On Wed, Nov 2, 2016 at 12:43 PM, Gerrit Jansen van Vuuren < > gerrit...@gmail.com> wrote: > > > Hi, > > > &

Re: Question regarding dynamic subscriber environment

Hi, Have a look at the kafka client lib https://github.com/gerritjvv/kafka-fast#java-1, it already provides this functionality. On Wed, Nov 2, 2016 at 2:34 AM, Janagan Sivagnanasundaram < janagan1...@gmail.com> wrote: > Kafka's current nature is does not support to dynamic subscriber >

Re: Topic with many paritions and many consumers, recommendations wanted.

take a look at kafka client https://github.com/gerritjvv/kafka-fast, it uses a different approach where you can have more than several consumers per topic+partition (i.e no relation between topic partitions and consumers). It uses redis but only for offsets and work distribution, not for the

Re: G1 tuning

Hi, I've seen pauses using G1 in other applications and have found that -XX:+UseParallelGC -XX:+UseParallelOldGC works best if you're having GC issues in general on the JVM. Regards, Gerrit On Wed, Oct 14, 2015 at 4:28 PM, Cory Kolbeck wrote: > Hi folks, > > I'm a bit

Re: automatically consume from all topics

Hi, I'm not sure about the high level consumer but I maintain a kafka consumer that can add and remove topics dynamically. https://github.com/gerritjvv/kafka-fast see https://github.com/gerritjvv/kafka-fast/blob/master/kafka-clj/java/kakfa_clj/core/Consumer.java if you're using java/scala On

Re: kafka java api (written in 100% clojure)

library under failures, particularly when Redis fails? -- Daniel On 13/10/2014, at 10:22 am, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: Hi, Just thought I'll put this out for the kafka community to see (if anyone finds it useful great!!). Kafka-fast is 100% pure

Re: kafka java api (written in 100% clojure)

as: • A lock service • A queue • A database On 13/10/2014, at 10:22 am, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: Hi, Just thought I'll put this out for the kafka community to see (if anyone finds it useful great!!). Kafka-fast is 100% pure

Re: kafka java api (written in 100% clojure)

of curiosity: did you choose Redis because ZooKeeper is not well supported in Clojure? Or were there other reasons? On Mon, Oct 13, 2014 at 2:04 PM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: Hi Steven, Redis: I've had a discussion on redis today, and one architecture that does

kafka java api (written in 100% clojure)

Hi, Just thought I'll put this out for the kafka community to see (if anyone finds it useful great!!). Kafka-fast is 100% pure clojure implementation for kafka, but not just meant for clojure because it has a Java API wrapper that can be used from Java, Groovy, JRuby or Scala. This library does

cannot replicate topics kafka inconsistent state

Hi, I've installed kafka 2.8.1, created a topic using: /opt/kafka/bin/kafka-topics.sh --create --topic test --zookeeper localhost:2381 --partitions 2 --replication-factor 2 Then opened a console producer and a console consumer. I type a few lines on the producer and then the two kafka brokers

Re: cannot replicate topics kafka inconsistent state

, Jun 17, 2014 at 1:48 AM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: Hi, I've installed kafka 2.8.1, created a topic using: /opt/kafka/bin/kafka-topics.sh --create --topic test --zookeeper localhost:2381 --partitions 2 --replication-factor 2 Then opened a console

kafka.common.LeaderNotAvailableException

Hi, I'm testing kafka 0.8.0 failover. I have 5 brokers 1,2,3,4,5. I shutdown 5 (with controlled shutdown activated). broker 4 is my bootstrap broker. My config has: default.replication.factor=2, num.partitions=8. When I look at the kafka server.log on broker 4 I get the below error, which only

Re: custom kafka consumer - strangeness

Hi, I've finally fixed this by closing the connection on timeout and creating a new connection on the next send. Thanks, Gerrit On Tue, Jan 14, 2014 at 10:20 AM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: Hi, thanks I will do this. On Tue, Jan 14, 2014 at 9:51 AM, Joe Stein

Re: kafka.common.LeaderNotAvailableException

I've found the response to my own question: http://mail-archives.apache.org/mod_mbox/kafka-users/201308.mbox/%3c44d1e1522419a14482f89ff4ce322ede25025...@brn1wnexmbx01.vcorp.ad.vrsn.com%3E On Wed, Jan 29, 2014 at 1:17 PM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: Hi, I'm testing

Re: custom kafka consumer - strangeness

? Thanks, Jun On Mon, Jan 13, 2014 at 8:42 AM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: I'm using netty and async write, read. For read I used a timeout such that if I do not see anything on the read channel, my read function times out and returns null. I do not see any error

Re: custom kafka consumer - strangeness

Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Tue, Jan 14, 2014 at 3:38 AM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: Yes, I'm using my own client

Re: custom kafka consumer - strangeness

? If you haven't enabled compression, it's weird that you will re-get 240 and 241 with an offset of 242 in the fetch request. Is that easily reproducible? Thanks, Jun On Mon, Jan 13, 2014 at 1:26 AM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: Hi, the offset in g is 240

Re: custom kafka consumer - strangeness

is responsible for advancing the offsets after consumption. Thanks, Jun On Thu, Jan 9, 2014 at 1:00 PM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: Hi, I'm writing a custom consumer for kafka 0.8. Everything works except for the following: a. connect, send fetch, read all results

Re: custom kafka consumer - strangeness

: FetchRequest; Version: 0; CorrelationId: 1389443537; ClientId: 1; ReplicaId: -1; MaxWait: 1000 ms; MinBytes: 1 bytes; RequestInfo: [ping,0] - PartitionFetchInfo(187,1048576). This corresponds with the timed out fetch request. On Sat, Jan 11, 2014 at 12:19 PM, Gerrit Jansen van Vuuren gerrit

Re: Velocity on local machine

Have you tried using more producers. The kafka broker is performant, but the client producer's performance is not what it should be. You can also have a look at tuning the number of kafka broker's network and io threads. Regards, Gerrit On Fri, Jan 10, 2014 at 1:06 PM, Klaus Schaefers

custom kafka consumer - strangeness

Hi, I'm writing a custom consumer for kafka 0.8. Everything works except for the following: a. connect, send fetch, read all results b. send fetch c. send fetch d. send fetch e. via the console publisher, publish 2 messages f. send fetch :corr-id 1 g. read 2 messages published :offsets [10 11]

Re: custom kafka consumer - strangeness

of the compressed block. Thus a message we saw previously may be returned again. This is probably what is happening to you Chris On Thu, Jan 9, 2014 at 4:00 PM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: Hi, I'm writing a custom consumer for kafka 0.8. Everything works

Re: java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33

, which we don't do currently. Thanks, Jun On Wed, Jan 1, 2014 at 9:27 AM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: Mm... Could be Im not sure if in a single request though. I am moving allot of data. Any pointer at were in the code the overflow might start? On 1 Jan 2014

Re: java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33

at 2:42 AM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: Hi, I just double checked my configuration and the broker has message.max.bytes set to 1 gig, the consumers have the same setting for max fetch size. I've lowered this to 600 mb and still see the same error

Re: java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33

reduce the max message size? Do you really expect to have a single message of 600MB? After that, you can reduce the fetch size. Thanks, Jun On Thu, Jan 2, 2014 at 8:06 AM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: There is a particular topic that has allot of data in each

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

The producer is heavily synchronized (i.e. all the code in the send method is encapsulated in one huge synchronized block). Try creating multiple producers and round robin send over them. e.g. p = producers[ n++ % producers.length ] p.send msg This will give you one thread per producer

java.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33

While consuming from the topics I get an IlegalArgumentException and all consumption stops, the error keeps on throwing. I've tracked it down to FectchResponse.scala line 33 The error happens when the FetchResponsePartitionData object's readFrom method calls:

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

1, 2014 at 2:22 PM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: The producer is heavily synchronized (i.e. all the code in the send method is encapsulated in one huge synchronized block). Try creating multiple producers and round robin send over them. e.g. p = producers[ n

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

it is not as explained here: http://engineering.gnip.com/kafka-async-producer/ On Wed, Jan 1, 2014 at 2:39 PM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: I don't know the code enough to comment on that (maybe someone else on the user list can do that), but from what I've

Re: Kafka producer behavior

, 2013 at 5:59 PM, Guozhang Wang wangg...@gmail.com wrote: Hello, This issue is known as in this JIRA: https://issues.apache.org/jira/browse/KAFKA-1067 Guozhang On Tue, Dec 17, 2013 at 8:48 AM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: hi, I've had