New broker not receiving any data

2013-04-29 Thread Matthew Rathbone
se? I presume it's fixed in 0.8. -- Matthew Rathbone Foursquare | Software Engineer | Server Engineering Team matt...@foursquare.com | @rathboma <http://twitter.com/rathboma> | 4sq<http://foursquare.com/rathboma>

Re: We have an invalid message in a segment, how to deal with it?

2013-04-13 Thread Matthew Rathbone
't been performed the broker should > > run a recovery procedure on the log that includes checksumming all > > messages. Invalid messages will be removed. Sounds like this isn't > > what happened? > > > > -Jay > > > > On Sat, Apr 13, 2013 at 10:30 AM

We have an invalid message in a segment, how to deal with it?

2013-04-13 Thread Matthew Rathbone
whole segment causes the broker to fail on startup with an error: java.lang.IllegalStateException: The following segments don't validate: , I'm happy losing that file if needs be, but I need to get this broker back up asap. -- Matthew Rathbone Foursquare | Software Engineer | Server E

We have an invalid message on a broker, how to deal with it?

2013-04-13 Thread Matthew Rathbone
bad segment causes the broker to fail on startup with an error: java.lang.IllegalStateException: The following segments don't validate: , I'm happy losing that file if needs be, but I need to get this broker back up asap. -- Matthew Rathbone Foursquare | Software Engineer | Server E

Re: .net (c#) kafka client

2013-04-03 Thread Matthew Rathbone
) > >>> but afaik, none for 0.8. > >>> > >>> Re: rest API: it is one of the current open projects that people can > >>> contribute to. There have been some discussions on the mailing list > about > >>> implementing one and a jira as well: KA

Re: Limit on number of Kafka Topic

2013-03-27 Thread Matthew Rathbone
> Wanted to check if there is any known limit on the # of topics in a > > Kafka > > >> cluster? I wanted to design a system which has say 5k topics and > > >> multi-threaded consumers reading messages from these topics. Does > anyone > > >> have

Re: Is anyone able to consume from Kafka 0.7.x and write into Hadoop CDH 4.x ?

2013-03-15 Thread Matthew Rathbone
;> destroy and notify the sender. Any use of this email is prohibited when >> received in error. Impetus does not represent, warrant and/or guarantee, >> that the integrity of this communication has been maintained nor that the >> communication is free of errors, virus, intercep

Kafka talk tomorrow night in Madison WI

2013-02-25 Thread Matthew Rathbone
Hey all, If any of you are in the midwest I'll get giving a talk on Kafka at the BigDataMadison meetup [1] in Madison WI. Stop by if you're around, there will be pizza :-). It's at 6pm tomorrow (feb 26th). [1]: http://www.meetup.com/BigDataMadison/ -- Matthew Rathbone Foursq

Re: S3 Consumer

2012-12-28 Thread Matthew Rathbone
e all the received > >> events to S3 bucket (say every minute). Something similar to Flume > >> HDFSSink > >> http://flume.apache.org/**FlumeUserGuide.html#hdfs-sink< > http://flume.apache.org/FlumeUserGuide.html#hdfs-sink> > >> I have tried evaluat