Yeah also the integration tests need to be specifically run - I would have thought the contributor would have run those tests and also tested the change themselves using live Kinesis :(
— Sent from Mailbox On Fri, Dec 11, 2015 at 6:18 AM, Burak Yavuz <brk...@gmail.com> wrote: > I don't think the Kinesis tests specifically ran when that was merged into > 1.5.2 :( > https://github.com/apache/spark/pull/8957 > https://github.com/apache/spark/commit/883bd8fccf83aae7a2a847c9a6ca129fac86e6a3 > AFAIK pom changes don't trigger the Kinesis tests. > Burak > On Thu, Dec 10, 2015 at 8:09 PM, Nick Pentreath <nick.pentre...@gmail.com> > wrote: >> Yup also works for me on master branch as I've been testing DynamoDB >> Streams integration. In fact works with latest KCL 1.6.1 also which I was >> using. >> >> So theKCL version does seem like it could be the issue - somewhere along >> the line an exception must be getting swallowed. Though the tests should >> have picked this up? Will dig deeper. >> >> — >> Sent from Mailbox <https://www.dropbox.com/mailbox> >> >> >> On Thu, Dec 10, 2015 at 11:07 PM, Brian London <brianmlon...@gmail.com> >> wrote: >> >>> Yes, it worked in the 1.6 branch as of commit >>> db5165246f2888537dd0f3d4c5a515875c7358ed. That makes it much less >>> serious of an issue, although it would be nice to know what the root cause >>> is to avoid a regression. >>> >>> On Thu, Dec 10, 2015 at 4:03 PM Burak Yavuz <brk...@gmail.com> wrote: >>> >>>> I've noticed this happening when there was some dependency conflicts, >>>> and it is super hard to debug. >>>> It seems that the KinesisClientLibrary version in Spark 1.5.2 is 1.3.0, >>>> but it is 1.2.1 in Spark 1.5.1. >>>> I feel like that seems to be the problem... >>>> >>>> Brian, did you verify that it works with the 1.6.0 branch? >>>> >>>> Thanks, >>>> Burak >>>> >>>> On Thu, Dec 10, 2015 at 11:45 AM, Brian London <brianmlon...@gmail.com> >>>> wrote: >>>> >>>>> Nick's symptoms sound identical to mine. I should mention that I just >>>>> pulled the latest version from github and it seems to be working there. >>>>> To >>>>> reproduce: >>>>> >>>>> >>>>> 1. Download spark 1.5.2 from http://spark.apache.org/downloads.html >>>>> 2. build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -DskipTests >>>>> clean package >>>>> 3. build/mvn -Pkinesis-asl -DskipTests clean package >>>>> 4. Then run simultaneously: >>>>> 1. bin/run-example streaming.KinesisWordCountASL [Kinesis app name] >>>>> [Kinesis stream name] [endpoint URL] >>>>> 2. bin/run-example streaming.KinesisWordProducerASL [Kinesis >>>>> stream name] [endpoint URL] 100 10 >>>>> >>>>> >>>>> On Thu, Dec 10, 2015 at 2:05 PM Jean-Baptiste Onofré <j...@nanthrax.net> >>>>> wrote: >>>>> >>>>>> Hi Nick, >>>>>> >>>>>> Just to be sure: don't you see some ClassCastException in the log ? >>>>>> >>>>>> Thanks, >>>>>> Regards >>>>>> JB >>>>>> >>>>>> On 12/10/2015 07:56 PM, Nick Pentreath wrote: >>>>>> > Could you provide an example / test case and more detail on what >>>>>> issue >>>>>> > you're facing? >>>>>> > >>>>>> > I've just tested a simple program reading from a dev Kinesis stream >>>>>> and >>>>>> > using stream.print() to show the records, and it works under 1.5.1 >>>>>> but >>>>>> > doesn't appear to be working under 1.5.2. >>>>>> > >>>>>> > UI for 1.5.2: >>>>>> > >>>>>> > Inline image 1 >>>>>> > >>>>>> > UI for 1.5.1: >>>>>> > >>>>>> > Inline image 2 >>>>>> > >>>>>> > On Thu, Dec 10, 2015 at 5:50 PM, Brian London < >>>>>> brianmlon...@gmail.com >>>>>> > <mailto:brianmlon...@gmail.com>> wrote: >>>>>> > >>>>>> > Has anyone managed to run the Kinesis demo in Spark 1.5.2? The >>>>>> > Kinesis ASL that ships with 1.5.2 appears to not work for me >>>>>> > although 1.5.1 is fine. I spent some time with Amazon earlier in >>>>>> the >>>>>> > week and the only thing we could do to make it work is to change >>>>>> the >>>>>> > version to 1.5.1. Can someone please attempt to reproduce >>>>>> before I >>>>>> > open a JIRA issue for it? >>>>>> > >>>>>> > >>>>>> >>>>>> -- >>>>>> Jean-Baptiste Onofré >>>>>> jbono...@apache.org >>>>>> http://blog.nanthrax.net >>>>>> Talend - http://www.talend.com >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>>> >>>>>> >>>> >>
