Re: [VOTE] Add REST Server to Apache Kafka
+1. On Tue, Oct 25, 2016 at 2:58 PM, Jay Kreps wrote: > -1 > > I think the REST server for Kafka that already exists is quite good and > getting contributions. Moving this into the core project doesn't solve a > problem that I see. > > -Jay > > On Tue, Oct 25, 2016 at 2:16 PM, Harsha Chintalapani > wrote: > > > Hi All, > >We are proposing to have a REST Server as part of Apache > Kafka > > to provide producer/consumer/admin APIs. We Strongly believe having > > REST server functionality with Apache Kafka will help a lot of users. > > Here is the KIP that Mani Kumar wrote > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 80:+Kafka+Rest+Server. > > There is a discussion thread in dev list that had differing opinions on > > whether to include REST server in Apache Kafka or not. You can read more > > about that in this thread > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201610.mbox/%3CCAMVt_ > > aymqeudm39znsxgktpdde46sowmqhsxop-+jmbcuv7...@mail.gmail.com%3E > > > > This is a VOTE thread to check interest in the community for > > adding REST Server implementation in Apache Kafka. > > > > Thanks, > > Harsha > > >
Re: How can i secure Kafka topics
If you are ok with SSL instead of kerberos follow this guide https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka The patch to support for kerberos just went in see https://issues.apache.org/jira/browse/KAFKA-1686. But the usage doc still needs to be created. On 10/23/15, 10:15 AM, "prateek arora" wrote: >Hi > >Can i secure Kafka topics like Apache hbase and hdfs have kerberos >security >mechanism ? > >i am not able to find out any document related to Kafka security . > > >please suggest how can i secure Kafka . > > > >Regards >Prateek
Re: OffsetOutOfRangeError with Kafka-Spark streaming
retention.ms is actually millisecond, you want a value much larger then 1440, which translates to 1.4 seconds. On 8/6/15, 4:35 PM, "Cassa L" wrote: >Hi Grant, >Yes, I saw exception in Spark and Kafka. In Kafka server logs I get this >exception: >kafka.common.OffsetOutOfRangeException: Request for offset 2823 but we >only >have log segments in the range 2824 to 2824. >at kafka.log.Log.read(Log.scala:380) >at >kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSet(KafkaApis.sc >ala:530) >at >kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1. >apply(KafkaApis.scala:476) >at >kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1. >apply(KafkaApis.scala:471) >at >scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scal >a:206) >at >scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scal >a:206) >at scala.collection.immutable.Map$Map1.foreach(Map.scala:105) >at >scala.collection.TraversableLike$class.map(TraversableLike.scala:206) >at scala.collection.immutable.Map$Map1.map(Map.scala:93) >at >kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSets(KafkaApis.s >cala:471) >at >kafka.server.KafkaApis$FetchRequestPurgatory.expire(KafkaApis.scala:783) >at >kafka.server.KafkaApis$FetchRequestPurgatory.expire(KafkaApis.scala:765) >at >kafka.server.RequestPurgatory$ExpiredRequestReaper.run(RequestPurgatory.sc >a > >Similar kind of exception comes to Spark Job. > >Here are my versions : > Spark - 1.4.1 >Kafka - 0.8.1 > >I changed retention on config using this command : >./kafka-topics.sh --alter --zookeeper XXX:2181 --topic MyTopic --config >retention.ms=1440 (I believe this is in minutes) > >I am also noticing something in Kafka. When I run below command on broker: >./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list >vdc-vm8.apple.com:9092 --topic MyTopic --time -2 >Earliest offset is being set to latest just in few seconds. Am I >co-relating this issue correctly? > >Here is my example on a new Topic. Initial output of this command is >./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list >vdc-vm8.apple.com:9092 --topic MyTopic --time -2 >SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". >SLF4J: Defaulting to no-operation (NOP) logger implementation >SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further >details. >MyTopic:0:60 > >I published 4 messages to Kafka. Immediately after few seconds, command >output is: > MyTopic:0:64 >Isn't this supposed to stay at 60 for longer time based on retention >policy? > > >Thanks, >Leena > > >On Thu, Aug 6, 2015 at 12:09 PM, Grant Henke wrote: > >> Does this Spark Jira match up with what you are seeing or sound related? >> https://issues.apache.org/jira/browse/SPARK-8474 >> >> What versions of Spark and Kafka are you using? Can you include more of >>the >> spark log? Any errors shown in the Kafka log? >> >> Thanks, >> Grant >> >> On Thu, Aug 6, 2015 at 1:17 PM, Cassa L wrote: >> >> > Hi, >> > Has anyone tried streaming API of Spark with Kafka? I am >>experimenting >> new >> > Spark API to read from Kafka. >> > KafkaUtils.createDirectStream(...) >> > >> > Every now and then, I get following error "spark >> > kafka.common.OffsetOutOfRangeException" and my spark script stops >> working. >> > I have simple topic with just one partition. >> > >> > I would appreciate any clues on how to debug this issue. >> > >> > Thanks, >> > LCassa >> > >> >> >> >> -- >> Grant Henke >> Software Engineer | Cloudera >> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke >>
Re: OffsetOutOfRangeError with Kafka-Spark streaming
In Apache Storm some users reported the same issue few months ago [1][2][3]. This was an unusual situation which in our experience only happened when storm topology was asking for offsets that were already trimmed by kafka. Multiple pathological cases(too low retention period, too slow topology, some poison pill message that kept retrying and finally got trimmed by kafka and was no longer available) because of which users may end up in that situation. Storm to an extent allows user to control what they want to do in this situation, not so sure about spark streaming. If spark does not handle this right now your best bet is to ensure your kafka retention period is high enough that your processing does not fall behind so much that the data gets trimmed (and this you should anyway do to avoid data loss) and ensure that you throw away/push to DLQ, failed messages after some small number of retries instead of retrying forever. [1]https://issues.apache.org/jira/browse/STORM-511 [2]https://issues.apache.org/jira/browse/STORM-586 [3]https://issues.apache.org/jira/browse/STORM-643 Thanks Parth On 8/6/15, 12:09 PM, "Grant Henke" wrote: >Does this Spark Jira match up with what you are seeing or sound related? >https://issues.apache.org/jira/browse/SPARK-8474 > >What versions of Spark and Kafka are you using? Can you include more of >the >spark log? Any errors shown in the Kafka log? > >Thanks, >Grant > >On Thu, Aug 6, 2015 at 1:17 PM, Cassa L wrote: > >> Hi, >> Has anyone tried streaming API of Spark with Kafka? I am experimenting >>new >> Spark API to read from Kafka. >> KafkaUtils.createDirectStream(...) >> >> Every now and then, I get following error "spark >> kafka.common.OffsetOutOfRangeException" and my spark script stops >>working. >> I have simple topic with just one partition. >> >> I would appreciate any clues on how to debug this issue. >> >> Thanks, >> LCassa >> > > > >-- >Grant Henke >Software Engineer | Cloudera >gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
Re: Access control in kafka
If this (https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+I nterface) is what you need then watch for https://reviews.apache.org/r/34492/ to get committed to trunk. Thanks Parth On 8/4/15, 1:57 PM, "Alvaro Gareppe" wrote: >Can someone point me to documentation about access control in kafka. There >is something implemented in the current or plan for future versions ? > >I need something that allows me to define what users are allowed to >connect >to certain topic, and of course user management. > >Thank you guys in advance! > >-- >Eng. Alvaro Gareppe
