Re: Kafka to transport binary files

Hi Rohit, librdkafka will produce messages with zero copy (unless compression is enabled). And kafkacat will do this if you provide it one or more files as arguments in producer mode: kafkacat -b broker -t topic -P your_file another_file the_thirdest_of_files (each file is sent as one message)

Plugable metadata store

Hello Apache Kafka community, Is it already possible to configure/use a different metadata store (topics, consumer groups, consumer to partition assignments, etc.) instead of ZooKeeper? If not, are there any plans to make it plugable in future? Kind regards, Stevo Slavic

Re: Plugable metadata store

I haven't been following closely but getting rid of zookeeper is in the pipeline. Look up 0.9 plans. They're somewhere on the wiki. Sent from my iPhone On 14-Nov-2014, at 5:18 pm, Stevo Slavić ssla...@gmail.com wrote: Hello Apache Kafka community, Is it already possible to configure/use

Re: Plugable metadata store

Isn't it only for offset management? Kind regards, Stevo On Fri, Nov 14, 2014 at 2:16 PM, Sharninder sharnin...@gmail.com wrote: I haven't been following closely but getting rid of zookeeper is in the pipeline. Look up 0.9 plans. They're somewhere on the wiki. Sent from my iPhone On

Kafka broker cannot start after running out of disk space

Hello, I've run into an issue with Kafka 0.8.1.1 broker. The broker stopped working after the disk it was writing to ran out of space. I freed up some space and tried to restart the broker. It started some recovery procedure, but after some short time in the logs I see the following strange error

Re: Programmatic Kafka version detection/extraction?

Mark, For non-Java clients an option could be to expose JMX via REST API using Jolokia as an adapter. This may be helpful: http://stackoverflow.com/questions/5106847/access-jmx-agents-from-non-java-clients Joel, I'm not familiar with Kafka build infrastructure, but e. g. Jenkins can easily

Re: Programmatic Kafka version detection/extraction?

Please let's not add dependencies on more third party protocols/software, the move away from Zookeeper dependence on the clients is very much welcomed and it'd be a pity to see a new foreign dependency added for such a trivial thing as propagating the version of a broker to its client. My

Re: Programmatic Kafka version detection/extraction?

I'm not sure there's a single protocol version. Each request/response has its own version ID embedded in the request itself. Broker version, broker ID and (as Joel suggested) git hash are all reasonable. And +1 for adding this to API to support non-JVM clients. Gwen On Fri, Nov 14, 2014 at 8:46

Re: Broker keeps rebalancing

We are using zookeeper-3.4.6 more zookeeper logs here: 2014-11-14 09:42:28,233 [myid:1] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for client /10.93.83.43:43487 which had sessionid 0x149a4cc1b581b60 2014-11-14 09:42:28,234 [myid:1] - WARN

Consumer reading from same topic twice without commiting

Hi, I am trying to understand the behavior of Kafka Consumer in Java. Consider a scenario where a consumer is reading from a partition with auto.comiit disabled and only one partition in the topic. The consumer reads a message 'A' and sleep for some time, say 10 seconds. Then reads from the same

Enforcing Network Bandwidth Quote with New Java Producer

HI Kafka Team, We like to enforce a network bandwidth quota limit per minute on producer side. How can I do this ? I need some way to count compressed bytes on producer ? I know there is callback does not give this ability ? Let me know the best way. Thanks, Bhavesh

Re: log4j dir?

Anyone has any idea how do I config the log4j file dir? On Thu, Nov 13, 2014 at 4:58 PM, hsy...@gmail.com hsy...@gmail.com wrote: Hi guys, Just notice kafka.logs.dir in log4j.properties doesn't take effect It's always set to *$base_dir/logs* in kafka-run-class.sh LOG_DIR=$base_dir/logs

Re: Programmatic Kafka version detection/extraction?

Hi Gwen, the protocol version in an InfoResponse message would be the maximum supported mainline protocol version, each change to the effective protocol specification bumps the version by one. The current per-request-type version ID is unfortunately useless to the client, it only allows the

Re: Offset manager movement (due to change in KAFKA-1469)

I just wanted to follow-up on a fall-out caused by the issue mentioned below. After the offset manager moved, consumer offsets started going to a different partition of the offsets topic. However, the previous partition of the offsets topic can still have the old offsets. Those won't necessarily

Re: log4j dir?

Hi, I had trouble with this as well. The version of Kafka I'm running insists on using 'kafka/logs', so I create a soft link from there to the desired destination directory: # kafka scripts hard-code the logs dir, so point that path to where we want the logs to be. ln -s $STREAM_BUFFER_LOGS_DIR

Re: Location of Logging Files/How To Turn On Logging For Kafka Components

You need to set the location of log4j.properties in a JVM variable. Take a look at the kafka scripts in bin/ Thanks, Jun On Thu, Nov 13, 2014 at 10:53 PM, Alex Melville amelvi...@g.hmc.edu wrote: Hi Jun, These are the two lines of log4j-related warnings I get when I try to run my producer:

Re: Getting Simple consumer details using MBean

So, you want to monitor the mbeans on the broker side? Take a look at http://kafka.apache.org/documentation.html#monitoring Thanks, Jun On Thu, Nov 13, 2014 at 10:58 PM, Madhukar Bharti bhartimadhu...@gmail.com wrote: Hi Jun Rao, Sorry to disturb you. But I my Kafka setup it is not showing.

Re: Consumer reading from same topic twice without commiting

Auto Commit is to sync offset to zookeeper periodically. Without auto commit, consumer still maintains an offset locally. It explains this behavior. To be able to rewind, you need to use SimpleConsumer. 2014/11/15 上午2:18 於 dinesh kumar dinesh...@gmail.com 寫道： Hi, I am trying to understand the

Re: Enforcing Network Bandwidth Quote with New Java Producer

We have a metric that measures the per-topic bytes send rate (after compression). You can get the values through the producer api. Thanks, Jun On Fri, Nov 14, 2014 at 10:34 AM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: HI Kafka Team, We like to enforce a network bandwidth quota

Re: log4j dir?

I think there is no way to specify different log location without modifying shell script. The *kafka.**logs.dir* in log4j.properties file is misleading On Fri, Nov 14, 2014 at 1:24 PM, Ben Drees ben.dr...@gmail.com wrote: Hi, I had trouble with this as well. The version of Kafka I'm running

shrinking isr for partitions followed by error.. (kafka 0.8.1.1)

I had deleted the topic sams_2014-10-30 successfully, and later on one of our kafka brokers (broker id 4) crashed, Now the topic is gone from zookeeper ls /brokers/topics (doesn't show the topic anymore) but I still see the zookeeper logs filling up with these messages.. : [2014-11-14

Re: shrinking isr for partitions followed by error.. (kafka 0.8.1.1)

correction server.log ( not zookeeper logs. ) On Fri, Nov 14, 2014 at 5:04 PM, Manish maa...@gmail.com wrote: I had deleted the topic sams_2014-10-30 successfully, and later on one of our kafka brokers (broker id 4) crashed, Now the topic is gone from zookeeper ls /brokers/topics