Kafka and spark integration

Any body know the end to end hadoop data flow which has Kafka - spark integration. I am primarily concerned on how kafka messages written to kafka partition gets captured into files?

Re: handleFetchRequest throw exception

Thanks to guozhang. According to your suggestions，I found my new patch to kafka 0.9.0.0 may casue the problem, In delayedfetch.scala, I include import org.apache.kafka.common.errors.NotLeaderForPartitionException but not import kafka.common.NotLeaderForPartitionException for intelij auto import

Re: handleFetchRequest throw exception

Json, As you mentioned yourself the "NotLeaderForPartitionException" thrown from getLeaderReplicaIfLocal should be caught in the end, and hence I'm not sure why the reported stack trace "ERROR: ..." throwing the NotLeaderForPartitionException should be seen from "tryComplete". Also I have checked

Re: kafka streaming rocks db lock bug?

Hello Ara, I have looked through the logs you sent to me (thanks!), and I'm suspecting the "file does not exist" issue is because your state directory is set to ` /tmp/kafka-streams` folder with the default value. As for the "LOCK: No locks available" and "Failed to lock the state directory" issue

Re: issue with custom processor flush to rocksdb store

Yes this is similar meaning it was all about KafkaStreams not started correctly in my spring app and NOT a bug in KafkaStreams. Inside the comments in the JIRA I have mentioned what I was doing wrong. These type of exceptions largely indicate kafka streams was not started correctly Thanks for you

Re: issue with custom processor flush to rocksdb store

Is it a similar report as https://issues.apache.org/jira/browse/KAFKA-4344? On Tue, Oct 25, 2016 at 2:43 PM, saiprasad mishra wrote: > Hi > This is with version 10.1.0 kafka streams (server running in remote and > streams app running local in my laptop). > > > > I have a kafka stream pipeline li

Re: [VOTE] Add REST Server to Apache Kafka

-1 for http kafka client in core Although a read only management interface, perhaps via http, sounds kinda useful for things like health checks as mentioned. On Wed, Oct 26, 2016 at 2:00 PM, Zakee wrote: > -1 > > Thanks. > > On Oct 25, 2016, at 2:16 PM, Harsha Chintalapani > wrote: > > > > Hi

Re: [VOTE] Add REST Server to Apache Kafka

-1 Thanks. > On Oct 25, 2016, at 2:16 PM, Harsha Chintalapani wrote: > > Hi All, > We are proposing to have a REST Server as part of Apache Kafka > to provide producer/consumer/admin APIs. We Strongly believe having > REST server functionality with Apache Kafka will help a lot of user

Re: [VOTE] Add REST Server to Apache Kafka

+1 supporting this natively will help immensely in building operational tools on top of it and augment ease of use cases in large deployments. -Guru From: Shekar Tippur Sent: Wednesday, October 26, 2016 9:23 AM To: users Cc: d...@kafka.apache.org Subject

Re: [VOTE] Add REST Server to Apache Kafka

-1 On Wed, Oct 26, 2016 at 9:23 AM, Shekar Tippur wrote: > +1 > > Thanks

Re: [VOTE] Add REST Server to Apache Kafka

+1 Thanks

Re: [VOTE] Add REST Server to Apache Kafka

I'm going to change my vote to a -1, great points made by all. (Although I would want to have another discussion around a "management" REST endpoint for Kafka, I think there would be value in that) On Wed, Oct 26, 2016 at 3:56 PM, Samuel Taylor wrote: > -1 > > I don't think the reasons from the

Re: [VOTE] Add REST Server to Apache Kafka

-1 I don't think the reasons from the KIP are good enough: 1 -- That other tools have a REST interface isn't a reason Kafka needs one. Further, "it is useful" is not a reason; *why* is it useful? 2 -- Why does a REST server *need* to be immediately available to every use that downloads Kafka? 3 -

Re: handleFetchRequest throw exception

it make the cluster can not provide normal service，which leades some producer or fetch fail for a long time before I restart current broker. this error may be come from some formerly fetch operation which contain this partition，which leads many fetch response error. The delayFetch's tryComplet

In-Reply-To=%3CCALsq4vKokx2_haKCwXHdoHrsVwn8UoBETCco6o1dwyFDoed1aw%40mail.gmail.com%3E&Subject=TimeoutException%3A%20failed%20to%20update%20metadata%20after

The KafkaProducer have a method partitionsFor() that fetches the metadata. So something like this can be used to force load the metadata with max.block.ms=0 while(!gotMetadata) { try { producer.partitionsFor(topic); gotMetadata=true; } catch (TimeoutException e) {

Re: Kafka multitenancy

Hi Mudit, This is a relevant read: http://www.confluent.io/blog/sharing-is-caring-multi-tenancy-in-distributed-data-systems It talks about two aspects of multi tenancy in Kafka, security and performance

Kafka multitenancy

Hi, How we can achieve multi-tenancy in kafka efficiently? Thanks,Mudit

Re: [VOTE] Add REST Server to Apache Kafka

And you also still need to find the correct broker, for each http call, wich is also hard, when programming against the http api On 26.10.2016 09:46, Jan Filipiak wrote: So happy to see this reply. I do think the same, actually makes it way harder to properly batch up records on http, as kafka

Re: [VOTE] Add REST Server to Apache Kafka

So happy to see this reply. I do think the same, actually makes it way harder to properly batch up records on http, as kafka core would need to know how to split your payload. It would help people do the wrong thing IMO best Jan On 25.10.2016 23:58, Jay Kreps wrote: -1 I think the REST serve