cc'ing dev list
Ok, looks like when the KCL version was updated in
https://github.com/apache/spark/pull/8957, the AWS SDK version was not,
probably leading to dependency conflict, though as Burak mentions its hard
to debug as no exceptions seem to get thrown... I've tested 1.5.2 locally
and on my
Yes, it's against master: https://github.com/apache/spark/pull/10256
I'll push the KCL version bump after my local tests finish.
On Fri, Dec 11, 2015 at 10:42 AM Nick Pentreath
wrote:
> Is that PR against master branch?
>
> S3 read comes from Hadoop / jet3t afaik
>
>
That's good news I've got a PR in to up the SDK version to 1.10.40 and the
KCL to 1.6.1 which I'm running tests on locally now.
Is the AWS SDK not used for reading/writing from S3 or do we get that for
free from the Hadoop dependencies?
On Fri, Dec 11, 2015 at 5:07 AM Nick Pentreath
Is that PR against master branch?
S3 read comes from Hadoop / jet3t afaik
—
Sent from Mailbox
On Fri, Dec 11, 2015 at 5:38 PM, Brian London
wrote:
> That's good news I've got a PR in to up the SDK version to 1.10.40 and the
> KCL to 1.6.1 which I'm running tests
Nick's symptoms sound identical to mine. I should mention that I just
pulled the latest version from github and it seems to be working there. To
reproduce:
1. Download spark 1.5.2 from http://spark.apache.org/downloads.html
2. build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0
Hi Nick,
Just to be sure: don't you see some ClassCastException in the log ?
Thanks,
Regards
JB
On 12/10/2015 07:56 PM, Nick Pentreath wrote:
Could you provide an example / test case and more detail on what issue
you're facing?
I've just tested a simple program reading from a dev Kinesis
Yes, it worked in the 1.6 branch as of commit
db5165246f2888537dd0f3d4c5a515875c7358ed. That makes it much less serious
of an issue, although it would be nice to know what the root cause is to
avoid a regression.
On Thu, Dec 10, 2015 at 4:03 PM Burak Yavuz wrote:
> I've
I've noticed this happening when there was some dependency conflicts, and
it is super hard to debug.
It seems that the KinesisClientLibrary version in Spark 1.5.2 is 1.3.0, but
it is 1.2.1 in Spark 1.5.1.
I feel like that seems to be the problem...
Brian, did you verify that it works with the
Yup also works for me on master branch as I've been testing DynamoDB Streams
integration. In fact works with latest KCL 1.6.1 also which I was using.
So theKCL version does seem like it could be the issue - somewhere along the
line an exception must be getting swallowed. Though the tests
I don't think the Kinesis tests specifically ran when that was merged into
1.5.2 :(
https://github.com/apache/spark/pull/8957
https://github.com/apache/spark/commit/883bd8fccf83aae7a2a847c9a6ca129fac86e6a3
AFAIK pom changes don't trigger the Kinesis tests.
Burak
On Thu, Dec 10, 2015 at 8:09 PM,
Has anyone managed to run the Kinesis demo in Spark 1.5.2? The Kinesis ASL
that ships with 1.5.2 appears to not work for me although 1.5.1 is fine. I
spent some time with Amazon earlier in the week and the only thing we could
do to make it work is to change the version to 1.5.1. Can someone
Yeah also the integration tests need to be specifically run - I would have
thought the contributor would have run those tests and also tested the change
themselves using live Kinesis :(
—
Sent from Mailbox
On Fri, Dec 11, 2015 at 6:18 AM, Burak Yavuz wrote:
> I don't
12 matches
Mail list logo