Enriched message metadata

2015-05-23 Thread Christopher Piggott
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Enriched+Message+Metadata Just curious if this proposal is in the development plan or not ... I'm particularly interested in being able to add custom KVP type metadata to messages. --Chris

Simple Consumer and offsets

2015-02-16 Thread Christopher Piggott
Hi, I am still using 0.8.1.1 because of the CPU use concerns. I'm confused about why the SimpleConsumer has: OffsetCommitResponse commitOffsets(OffsetCommitRequest request) and OffsetFetchResponse fetchOffsets(OffsetFetchRequest request) but no way that I can see to issue a ConsumerMetadataRe

Need sample/example of updating offset in SImple Consumer

2015-02-09 Thread Christopher Piggott
Hi, Can somebody provide me with an example of how to formulate an OffsetCommitRequest for a single stream/partition using SimpleConsumer from java? Both ends, really ... periodically issuing commits, but also how to get the current offset when starting up. I can show what I'm attempting ... bu

Re: Doubts Kafka

2015-02-08 Thread Christopher Piggott
e the client to have a callback mechanism for processing, and the marker only gets moved if the high level consumer successfully implements my callback/processor (with no exceptions, at least). On Sun, Feb 8, 2015 at 9:49 AM, Gwen Shapira wrote: > On Sun, Feb 8, 2015 at 6:39 AM, Christopher

Re: Doubts Kafka

2015-02-08 Thread Christopher Piggott
an in-memory view of the latest offset per > triplet, so offset fetch requests can be > served quickly without requiring a full scan of the compacted offsets > topic. With this feature, consumers can checkpoint offsets very often, > possibly per message. On Sun, Feb 8, 2015 at 9:39 AM,

Re: Doubts Kafka

2015-02-08 Thread Christopher Piggott
> The consumer used Zookeeper to store offsets, in 0.8.2 there's an option to use Kafka itself for that (by setting *offsets.storage = kafka Does it still really live in zookeeper, and kafka is proxying the requests through? On Sun, Feb 8, 2015 at 9:25 AM, Gwen Shapira wrote: > Hi Eduardo, > >

Re: Consumer questions

2015-01-17 Thread Christopher Piggott
est , earliest > or some other offset. > > 3. https://issues.apache.org/jira/browse/KAFKA-1779 > > 4. Yes > > > Manikumar > > On Sat, Jan 17, 2015 at 2:25 AM, Christopher Piggott > wrote: > > > Hi, > > > > I am following this link: > > > > >

Consumer questions

2015-01-16 Thread Christopher Piggott
messages, so presumably it fetches as many messages as will fit into that many byte buffer? Is that right? Thanks. Christopher Piggott Sr. Staff Engineer Golisano Institute for Sustainability Rochester Institute of Technology