Re: Kafka getMetadata api

You could do what you are asking with a custom encoder/decoder so the message bytes are made up of "messageType+AvroMessage" for the bytes of the message. The message can be whatever byte structure you want and not just the avro binary i.e. https://github.com/linkedin/camus/blob/master/camus-kafka-

Re: Kafka getMetadata api

Indeed my message size varies b/w ~500kb to ~5mb per avro. I am using kafka as a I need a scalable pub-sub messaging architecture with multiple produces and consumers and guaranty of delivery. Keeping data on filesystem or hdfs won't give me that. Also In the link below [1] there is a linkedin's

Re: kafka-web-console error

The kafka project doesn't have an official web console so you might need to open an issue on the github page of the project for the web console you are using as they may not be closing connections and using up all of resources regardless of what you have set, etc if you have the default setting

Re: kafka logs gone after reboot the server

Either should be fine but is all dependent on your environment and how you want to operate your cluster in the long run i.e. if you have multiple volumes then you need to have them each in the log.dirs separated by comma, etc. If there is no good reason to have the broker.id in the directory they y

kafka-web-console error

Hi, all I am running kafka-web-console, I periodically getting such error and cause the UI down: ! @6kldaf9lj - Internal server error, for (GET) [/assets/images/zookeeper_small.gif] -> play.api.Application$$anon$1: Execution exception[[FileNotFoundException: /vagrant/kafka-web-console-master/ta

Re: Kafka 0.8.1.1 Leadership changes are happening very often

We don¹t see zookeeper expiration. However I noticed that our servers system time is NOT synced. Hence server1 and server2 had 30+sec delay. Do you think that could cause leadership change or any other issue. On 12/31/14, 4:03 PM, "Jun Rao" wrote: >A typical cause of frequent leadership changes

Re: kafka logs gone after reboot the server

One more question, when I set the log.dirs in different nodes in the cluster, should I set them different name, say kafka-logs-1 which associated with broker id, or I can set the same directory name, like /var/log/kafka for every node (assume one broker in each server). thanks On Fri, Jan 2, 201

Re: kafka logs gone after reboot the server

Thanks a lot! On Fri, Jan 2, 2015 at 12:15 PM, Jay Kreps wrote: > Nice catch Joe--several people have complained about this as a problem and > we were a bit mystified as to what kind of bug could lead to all their logs > getting deleted and re-replicated when they bounced the server. We assumed

Re: kafka logs gone after reboot the server

Nice catch Joe--several people have complained about this as a problem and we were a bit mystified as to what kind of bug could lead to all their logs getting deleted and re-replicated when they bounced the server. We assumed "bounced" meant restarted the app, but I think likely what is happening i

Re: Kafka getMetadata api

Just wondering Mukesh - the reason you want this feature is because your value payload is not small (tens of kb). Don't know if that is the right usage of kafka. It might be worthwhile to store the avro files in a filesystem (regular, cluster fs, hdfs or even hbase) and the value in your kafka m

Re: Kafka getMetadata api

Thanks for your response guys, Filtering messages in the application works fine for me, I am just thinking out loud as fetching just the key will be much faster than fetching the entire Message&Metadata and will avoid unnecessary data transfer b/w kafka & consumer nodes, hence having an API to exp

Re: kafka logs gone after reboot the server

That is because your logs are in /tmp which you can change by setting log.dirs to something else. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

kafka logs gone after reboot the server

Hi, All I've just notice one thing, when I am experiencing some errors in Kafka servers, I reboot the dev servers (not a good way), after reboot, I get into zkCli, I can see all the topics still exist. But when I get into kafka log directory, I found all data gone, see root@DO-mq-dev:/tmp/kafka-l

Re: messages lost

Thanks a lot, Tim, this is the config of brokers -- broker.id=1 port=9092 host.name=10.100.70.128 num.network.threads=4 num.io.threads=8 socket.send.buffer.bytes=1048576 socket.receive.buffer.bytes=1048576 socket.request.max.bytes=104857600 auto.leader.rebalance.enable=true auto.create.top

Re: Kafka getMetadata api

I think partitioning is best left for the semantics of the message (i.e. userId, customerId, etc) and not the type of message. If your consumers only need specific message types then separate the messages types by topics. This will make the consumers that don't need those message types work better

Re: messages lost

What's your configured required.acks? And also are you waiting for all your messages to be acknowledged as well? The new producer returns futures back, but you still need to wait for the futures to complete. Tim On Fri, Jan 2, 2015 at 9:54 AM, Sa Li wrote: > Hi, all > > We are sending the messa

messages lost

Hi, all We are sending the message from a producer, we send 10 records, but we see only 99573 records for that topics, we confirm this by consume this topic and check the log size in kafka web console. Any ideas for the message lost, what is the reason to cause this? thanks -- Alec Li

Re: Kafka getMetadata api

Hi, One option is to partition the data using key and consume from relevant partition. Or your current approach (filtering messages in the application) should be OK. Using separate getMetaData/getkey and getMessage may hit the consumer performance/throughput. Regards, Kumar On Fri, Jan 2, 2015

Re: Kafka getMetadata api

Any pointers guys? On 1 Jan 2015 15:26, "Mukesh Jha" wrote: > Hello Experts, > > I'm using a kafka topic to store bunch of messages where the key contains > metadata and value is the data (avro file in our case). > There are multiple consumers for each topic and the consumer can decide if > the m