Re: Java Client for Kafka Consumer(0.9) not polling the records from brokers instantly

Don't think you cannot attach diagrams in these mailling lists. On Thu, May 26, 2016 at 7:35 PM, Navneet Kumar wrote: > Hi > We are facing a issue where our Consumer component is not instantly > logging the records in DB polled from the Brokers. We have following > theOur architecture diagram is

Re: Question about applicability of Kafka Streams

Thank you very much for the information! I'll look into it. Best regards KIM 2016-05-27 11:31 GMT+09:00 Tobias Adamson : > Hi Kim > Would maybe this example work for you? > > https://github.com/apache/kafka/tree/trunk/streams/examples/src/main/java/org/apache/kafka/streams/examples/pageview > <

Re: Java Client for Kafka Consumer(0.9) not polling the records from brokers instantly

Hi We are facing a issue where our Consumer component is not instantly logging the records in DB polled from the Brokers. We have following theOur architecture diagram is attached herewith. Following are the properties configured: Producer.properties bootstrap.servers=xx.xxx.xxx.140:9092,xx.xxx.x

Re: Question about applicability of Kafka Streams

Hi Kim Would maybe this example work for you? https://github.com/apache/kafka/tree/trunk/streams/examples/src/main/java/org/apache/kafka/streams/examples/pageview It included JS

Question about applicability of Kafka Streams

Hello. First I thank you so much for the devs since they've been making a great tool as open-source software. I'm considering to apply a new feature of the Kafka, aka Kafka Streams, on my simple handler application, which receives monitoring data from Collectd and reproduce transformed messages t

Re: Kafka Streams / Processor

Thank you. Some more follow-up questions 1) great, will do some tests 2) if auto commit is used how do we prevent a commit happening when an error happens in processing. Basically our scenario is that we build up aggregation contexts for specific keys (these are a bit special so most probably

Re: macbook air and kafka

Well... We added KafkaConnect and KafkaStreams, thats two fairly big features. On Thu, May 26, 2016 at 11:58 AM, S Ahmed wrote: > I just pulled lated on the same old 2010 MPB and the build took over 4 > minutes. > > Have things changed so much since 2013? :) > > I ran: ./gradlew jar > > On Tue,

Detecting connection loss to Kafka Broker with 0.9 Clients

I am trying to detect connection loss in my Kafka 0.9 Consumer and Producer so that I can check if I should update any properties (update passwords or truststore locations by restarting the client with new configs if they have been provided). I am having trouble finding good ways to detect if I ha

Failed to find leader for Set exception

Hi, I am using Apache NiFi to read data form Kafka which runs as part of Hortonworks HDP 2.4. I use GetKafka processor to read messages form a topic but see the exception bellow. What can be wrong? 2016-05-26 17:13:36,776 INFO [d217bceb-d58b-4031-9143-26d8ddd15e40_BPIW01a-1464297163195-b78e95a5-l

RE: FetchRequest Question

Ping -Original Message- From: Heath Ivie [mailto:hi...@autoanything.com] Sent: Wednesday, May 25, 2016 9:25 AM To: users@kafka.apache.org Subject: FetchRequest Question Can someone please explain why if I write 1 message to the queue it takes N FetchRequests to get the data out where n

Re: macbook air and kafka

I just pulled lated on the same old 2010 MPB and the build took over 4 minutes. Have things changed so much since 2013? :) I ran: ./gradlew jar On Tue, Jun 4, 2013 at 7:24 PM, Neha Narkhede wrote: > *Memory* 8 GB 1600 MHz DDR3 > > *Processor* 2 GHz Intel Core i7 > > > Thanks, > Neha > > > >

Re: Kafka Streams / Processor

Hi Toby, 1) I am not an expert for RocksDB, but I don't see a problem with larger objects. 2) I assume, by "guaranteed" you mean that the commit is performed when the call return. In this case, no. It only sets a flag to commit at the next earliest point in time possible. Ie, you can trigger comm

Kafka view msg tool - connection issues

Hi , I just install Kafka UI tooll http://www.kafkatool.com/download.html trying to view messages that going over on my server. I'm able to connect , but after a second, connection is clasoe with below exception? KeeperServer) [2016-05-26 18:05:16,535] INFO Client attempting to establish new

Kafka Streams / Processor

Hi We have a scenario where we could benefit from the new API’s instead of our in house ones. However we have a couple of questions 1. Is it feasible to save 2-3MB size values in the RocksDBStorage? 2. When is the offset committed as processed when using a custom Processor, is it when you call c

kafka connect - fetch avro data from the SinkRecord put method

Hi , I have been working on developing new Kafka connector ( Sink ) which pushes data to container from the kafka cluster. I am pushing the data using kafka-avro-console-producer to one of the topics . At the time of fetching the records pushed ( using SinkRecord ) at the put method in the SinkT

Re: Brokers changing mtime on data files during startup?

Hi! We noticed the same here with 0.9.0.1. To work around the issue a better way then to set a very low retention.ms is to set retention.bytes on a topic level, like this: ./bin/kafka-topics.sh --zookeeper X.X.X.X:2181/kafka -alter --config retention.bytes=500 –topic my_topic The settings

Re: Kafka encryption

> How can we do file system encryption? Google "LUKS" and you should be able to find some reasources on disk-level encryption. Cheers, Jens On Wed, May 25, 2016 at 11:59 AM Tom Crayford wrote: > If you're using EBS then it's a single flag to use encrypted drives at the > provision time of the

Re: Kafka broker crash - broker id then changed

Hi Ben, Thanks for responding. I can't imagine what would have cleaned temp up at that time. I don't think we have anything in place to do that, it also appears to happened to both machines at the same time. It also appears that the other topics were not affected, there were still other files pre

Re: Kafka broker crash - broker id then changed

Possibly tmp got cleaned up? Seems like one of the log files where deleted while a producer was writing messages to it: On Thursday, 26 May 2016, cs user wrote: > Hi All, > > We are running Kafka version 0.9.0.1, at the time the brokers crashed > yesterday we were running in a 2 mode cluster. T

Kafka broker crash - broker id then changed

Hi All, We are running Kafka version 0.9.0.1, at the time the brokers crashed yesterday we were running in a 2 mode cluster. This has now been increased to 3. We are not specifying a broker id and relying on kafka generating one. After the brokers crashed (at exactly the same time) we left kafka