Re: Format of Kafka storage on disk

2014-01-03 Thread Joe Stein
The DumpLogSegments should do that for you https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/DumpLogSegments.scala bin/kafka-run-class.sh kafka.tools.DumpLogSegments Option Description -- --- --

Format of Kafka storage on disk

2014-01-03 Thread Subbu Srinivasan
Is there any place where I can know about the internal structure of the log file where kafka stores the data. A topic has a .index and a .log file. I want to read the entire log file and parse the contents out. Thanks Subbu

Re: node.js client library?

2014-01-03 Thread Thomas
Thanks On 27/12/2013 18:49, Joe Stein wrote: I added the wurstmeister client to the wiki SOHU-Co, can you provide a license file in the project and I would link it then too please. https://cwiki.apache.org/confluence/display/KAFKA/Clients I also added wurstmeister's port of storm-kafka for 0.

Re: problem with high-level consumer stream filter regex....

2014-01-03 Thread Jason Rosenberg
Thanks Joe, I can confirm that your patch works for me, as applied to 0.8.0. Jason On Fri, Dec 20, 2013 at 6:28 PM, Jason Rosenberg wrote: > Thanks Joe, > > I generally build locally, and upload to our maven proxy (using a custom > pom). > > I haven't yet had luck using maven central (although,

Re: Trouble recovering after a crashed broker

2014-01-03 Thread Jun Rao
If a broker crashes and restarts, it will catch up the missing data from the leader replicas. Normally, when this broker is catching up, it won't be serving any client requests though. Are you seeing those errors on the crashed broker? Also, you are not supposed to see OffsetOutOfRangeException wit

Trouble recovering after a crashed broker

2014-01-03 Thread Vincent Rischmann
Hi all, We have a cluster of 3 0.8 brokers, and this morning one of the broker crashed. It is a test broker, and we stored the logs in /tmp/kafka-logs. All topics in use are replicated on the three brokers. You can guess the problem, when the broker rebooted it wiped all the data in the logs. Th