I don't think the Kinesis tests specifically ran when that was merged into 1.5.2 :( https://github.com/apache/spark/pull/8957 https://github.com/apache/spark/commit/883bd8fccf83aae7a2a847c9a6ca129fac86e6a3
AFAIK pom changes don't trigger the Kinesis tests. Burak On Thu, Dec 10, 2015 at 8:09 PM, Nick Pentreath <nick.pentre...@gmail.com> wrote: > Yup also works for me on master branch as I've been testing DynamoDB > Streams integration. In fact works with latest KCL 1.6.1 also which I was > using. > > So theKCL version does seem like it could be the issue - somewhere along > the line an exception must be getting swallowed. Though the tests should > have picked this up? Will dig deeper. > > — > Sent from Mailbox <https://www.dropbox.com/mailbox> > > > On Thu, Dec 10, 2015 at 11:07 PM, Brian London <brianmlon...@gmail.com> > wrote: > >> Yes, it worked in the 1.6 branch as of commit >> db5165246f2888537dd0f3d4c5a515875c7358ed. That makes it much less >> serious of an issue, although it would be nice to know what the root cause >> is to avoid a regression. >> >> On Thu, Dec 10, 2015 at 4:03 PM Burak Yavuz <brk...@gmail.com> wrote: >> >>> I've noticed this happening when there was some dependency conflicts, >>> and it is super hard to debug. >>> It seems that the KinesisClientLibrary version in Spark 1.5.2 is 1.3.0, >>> but it is 1.2.1 in Spark 1.5.1. >>> I feel like that seems to be the problem... >>> >>> Brian, did you verify that it works with the 1.6.0 branch? >>> >>> Thanks, >>> Burak >>> >>> On Thu, Dec 10, 2015 at 11:45 AM, Brian London <brianmlon...@gmail.com> >>> wrote: >>> >>>> Nick's symptoms sound identical to mine. I should mention that I just >>>> pulled the latest version from github and it seems to be working there. To >>>> reproduce: >>>> >>>> >>>> 1. Download spark 1.5.2 from http://spark.apache.org/downloads.html >>>> 2. build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -DskipTests >>>> clean package >>>> 3. build/mvn -Pkinesis-asl -DskipTests clean package >>>> 4. Then run simultaneously: >>>> 1. bin/run-example streaming.KinesisWordCountASL [Kinesis app name] >>>> [Kinesis stream name] [endpoint URL] >>>> 2. bin/run-example streaming.KinesisWordProducerASL [Kinesis >>>> stream name] [endpoint URL] 100 10 >>>> >>>> >>>> On Thu, Dec 10, 2015 at 2:05 PM Jean-Baptiste Onofré <j...@nanthrax.net> >>>> wrote: >>>> >>>>> Hi Nick, >>>>> >>>>> Just to be sure: don't you see some ClassCastException in the log ? >>>>> >>>>> Thanks, >>>>> Regards >>>>> JB >>>>> >>>>> On 12/10/2015 07:56 PM, Nick Pentreath wrote: >>>>> > Could you provide an example / test case and more detail on what >>>>> issue >>>>> > you're facing? >>>>> > >>>>> > I've just tested a simple program reading from a dev Kinesis stream >>>>> and >>>>> > using stream.print() to show the records, and it works under 1.5.1 >>>>> but >>>>> > doesn't appear to be working under 1.5.2. >>>>> > >>>>> > UI for 1.5.2: >>>>> > >>>>> > Inline image 1 >>>>> > >>>>> > UI for 1.5.1: >>>>> > >>>>> > Inline image 2 >>>>> > >>>>> > On Thu, Dec 10, 2015 at 5:50 PM, Brian London < >>>>> brianmlon...@gmail.com >>>>> > <mailto:brianmlon...@gmail.com>> wrote: >>>>> > >>>>> > Has anyone managed to run the Kinesis demo in Spark 1.5.2? The >>>>> > Kinesis ASL that ships with 1.5.2 appears to not work for me >>>>> > although 1.5.1 is fine. I spent some time with Amazon earlier in >>>>> the >>>>> > week and the only thing we could do to make it work is to change >>>>> the >>>>> > version to 1.5.1. Can someone please attempt to reproduce >>>>> before I >>>>> > open a JIRA issue for it? >>>>> > >>>>> > >>>>> >>>>> -- >>>>> Jean-Baptiste Onofré >>>>> jbono...@apache.org >>>>> http://blog.nanthrax.net >>>>> Talend - http://www.talend.com >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>> >>>>> >>> >
