Yes. data was flowing in. Do you see any exceptions? On Tue, May 3, 2016 at 6:11 PM, John Yost <[email protected]> wrote:
> Hey Abhishek, > > Cool, yeah, switched my kafka dependency (my topo reads and writes to > Kafka) to 0.8.2.1 and everything deploys correctly. The next issue I am > encountering is no data is flowing through the KafkaSpout. Do you have > data flowing in your topo? > > --John > > On Mon, May 2, 2016 at 7:17 AM, Abhishek Agarwal <[email protected]> > wrote: > >> good thing is that storm-kafka artifact itself is compatible with kafka >> 0.8.2.1. So in your maven project, you can simply exclude the >> org.apache.kafka dependencies coming out of storm-kafka project and use the >> version you need. >> >> On Mon, May 2, 2016 at 3:03 PM, Abhishek Agarwal <[email protected]> >> wrote: >> >>> John, I think you have hit it right. I started using storm-kafka 1.0 >>> with kafka server 0.8.2.1 and started running into all sorts of issues >>> including the one you pointed out. Also 0.9 documentation clearly states >>> that upgraded clients will not be compatible with older kafka version. >>> http://kafka.apache.org/090/documentation.html#upgrade >>> >>> On Wed, Apr 20, 2016 at 5:42 PM, John Yost <[email protected]> wrote: >>> >>>> Argh, fat fingers...I am attempting to write to Kafka 0.8.2.1 from >>>> Storm 1.0.0 which is has a dependency upon Kafka 0.9.0.1. >>>> >>>> @Abhishek -> interesting you are seeing the same exception for Storm >>>> 0.10.0 because that has a dependency upon Kafka 0.8.1.1. >>>> >>>> On Wed, Apr 20, 2016 at 8:06 AM, John Yost <[email protected]> >>>> wrote: >>>> >>>>> Oh, gotcha, okay, will do. BTW, here's the link I failed to provide >>>>> the first time: https://github.com/confluentinc/examples/issues/15 >>>>> >>>>> --John >>>>> >>>>> On Wed, Apr 20, 2016 at 7:44 AM, Abhishek Agarwal < >>>>> [email protected]> wrote: >>>>> >>>>>> @John - >>>>>> can you file a JIRA for this? I doubt it is related to 1.0.0 version >>>>>> in particular. I have run into "illegalArugmentExceptions" in KafkaSpout >>>>>> (0.10.0). >>>>>> >>>>>> On Wed, Apr 20, 2016 at 4:44 PM, John Yost <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Also, I found this link that indicates the exception I reported >>>>>>> yesterday can be symptomatic of a mismatch between the client and broker >>>>>>> where the client is one version newer. I am not saying that's the case >>>>>>> here with Storm 1.0.0, but wanted to provide this info >>>>>>> troubleshooting-wise. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> --John >>>>>>> >>>>>>> On Tue, Apr 19, 2016 at 3:26 PM, John Yost <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Harsha, >>>>>>>> >>>>>>>> When the Storm 1.0.0 KafkaSpout (from the storm-kafka jar) attempts >>>>>>>> to read from the Kafka 0.8.2.1 partition an IlegalArgumentException is >>>>>>>> thrown, the root exception of which is as follows: >>>>>>>> >>>>>>>> at java.nio.Buffer.limit(Buffer.java:267) >>>>>>>> at >>>>>>>> kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:37) >>>>>>>> at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:99) >>>>>>>> at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:97) >>>>>>>> at >>>>>>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >>>>>>>> at >>>>>>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >>>>>>>> at scala.collection.immutable.Range.foreach(Range.scala:141) >>>>>>>> at >>>>>>>> scala.collection.TraversableLike$class.map(TraversableLike.scala:244) >>>>>>>> at scala.collection.AbstractTraversable.map(Traversable.scala:105) >>>>>>>> at kafka.api.TopicData$.readFrom(FetchResponse.scala:97) >>>>>>>> at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169) >>>>>>>> at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:168) >>>>>>>> at >>>>>>>> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) >>>>>>>> at >>>>>>>> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) >>>>>>>> at scala.collection.immutable.Range.foreach(Range.scala:141) >>>>>>>> at >>>>>>>> scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) >>>>>>>> at >>>>>>>> scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) >>>>>>>> >>>>>>>> The corresponding source code in Kafka where the root exception is >>>>>>>> thrown is bolded: >>>>>>>> >>>>>>>> object FetchResponsePartitionData { >>>>>>>> def readFrom(buffer: ByteBuffer): FetchResponsePartitionData = { >>>>>>>> val error = buffer.getShort >>>>>>>> val hw = buffer.getLong >>>>>>>> val messageSetSize = buffer.getInt >>>>>>>> val messageSetBuffer = buffer.slice() >>>>>>>> * messageSetBuffer.limit(messageSetSize)* >>>>>>>> buffer.position(buffer.position + messageSetSize) >>>>>>>> new FetchResponsePartitionData(error, hw, new >>>>>>>> ByteBufferMessageSet(messageSetBuffer)) >>>>>>>> } >>>>>>>> >>>>>>>> I am using all the default KafkaConfig settings for the KafkaSpout >>>>>>>> with the exception of startOffsetTime, so I don't *think* I have a >>>>>>>> misconfiguration, but I may be wrong. >>>>>>>> >>>>>>>> Please confirm if there is anything I need to do config-wise to >>>>>>>> make this work. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> --John >>>>>>>> >>>>>>>> On Sat, Apr 16, 2016 at 10:49 PM, <[email protected]> wrote: >>>>>>>> >>>>>>>>> Awesome, thanks Harsha! >>>>>>>>> >>>>>>>>> --John >>>>>>>>> >>>>>>>>> Sent from my iPhone >>>>>>>>> >>>>>>>>> > On Apr 16, 2016, at 1:28 PM, Harsha <[email protected]> wrote: >>>>>>>>> > >>>>>>>>> > John, >>>>>>>>> > I think you are asking if you will be able to run >>>>>>>>> 0.8.2 kafka consumer in storm 1.0.0 . Yes we are shipping >>>>>>>>> storm-kafka-client which uses the new consumer api in kafka 0.9.0.1 >>>>>>>>> but >>>>>>>>> storm 1.0.0 still ships with storm-kafka which uses older consumer api >>>>>>>>> which can work with all versions of kafka including 0.9.0.1. >>>>>>>>> > >>>>>>>>> > "I checked out the v1.0.0 tag, changed the kafka version to >>>>>>>>> 0.8.2.1, and I am getting compile errors in storm-kafka-client. I >>>>>>>>> don't >>>>>>>>> have a problem fixing the errors, but I want to ensure I am not >>>>>>>>> wasting my >>>>>>>>> time. :)" >>>>>>>>> > >>>>>>>>> > You shouldn't be changing kafka version. As I said above Kafka >>>>>>>>> 0.9.0.1 contains two kafka apis new ones which will only works with >>>>>>>>> 0.9.0.1 >>>>>>>>> kafka cluster and old consumer apis which can work 0.8.2. Even though >>>>>>>>> you >>>>>>>>> compile with 0.9.0.1 version it will work with 0.8.2.1 kafka cluster. >>>>>>>>> > >>>>>>>>> > Let me know if you have any questions. >>>>>>>>> > >>>>>>>>> > Thanks, >>>>>>>>> > Harsha >>>>>>>>> > >>>>>>>>> >> On Fri, Apr 15, 2016, at 11:39 AM, John Yost wrote: >>>>>>>>> >> Hi Everyone, >>>>>>>>> >> I know that Storm 1.0.0 is designed to work with Kafka 0.9.0.1, >>>>>>>>> but is it possible for Storm 1.0.0 to work with 0.8.2.1? >>>>>>>>> >> >>>>>>>>> >> I did some integration testing with 0.9.0.1 client with 0.8.2.1 >>>>>>>>> broker, which is yucky, and it appears this won't work. >>>>>>>>> >> >>>>>>>>> >> I checked out the v1.0.0 tag, changed the kafka version to >>>>>>>>> 0.8.2.1, and I am getting compile errors in storm-kafka-client. I >>>>>>>>> don't >>>>>>>>> have a problem fixing the errors, but I want to ensure I am not >>>>>>>>> wasting my >>>>>>>>> time. :) >>>>>>>>> >> >>>>>>>>> >> Please confirm if it is anticipated that Storm 1.0.0 can be >>>>>>>>> made compatible with Kafka 0.8.2.x--thanks! >>>>>>>>> >> --John >>>>>>>>> > >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> Abhishek Agarwal >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> Regards, >>> Abhishek Agarwal >>> >>> >> >> >> -- >> Regards, >> Abhishek Agarwal >> >> > -- Regards, Abhishek Agarwal
