Github user cowtowncoder commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-176310354
@srowen Ah. Yes, just wanted to mention it; I don't have enough context so
there may be specific reasons for picking up particular versions. Glad to see
version up
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-176305142
I think that's OK. I was intending to propose a series of dependency
updates for 2.x like this sort of thing.
---
If your project is set up for it, you can reply to th
Github user cowtowncoder commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-176300804
I know this issue is closed, but one suggestion I have would be to specify
the latest Jackson 2.5 version, 2.5.5 instead of 2.5.3. While I understand
caution in ge
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10256
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is ena
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-170484935
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this featu
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-170346366
I'm going to go for this one since I think we may want to, in general,
consider updating Spark dependencies in 2.x anyway. 2.5 is even old for
Jackson, so good to bump u
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-170341303
**[Test build #2358 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2358/consoleFull)**
for PR 10256 at commit
[`a1f75c1`](https://
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-170333530
**[Test build #2358 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2358/consoleFull)**
for PR 10256 at commit
[`a1f75c1`](https://g
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-170333472
@BrianLondon it should be running `dev/run-tests`, so that's something you
can inspect or run locally if you need to. The doc there describes how to run
one test.
---
Github user BrianLondon commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-170270526
Manifest updated. Is there documented somewhere what all the tests Travis
runs are? It seems to differ from what's described at
http://spark.apache.org/docs/lates
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-170116355
Ah right, you'll need to acknowledge the dep changes in the build file:
```
To update the manifest file, run './dev/test-dependencies.sh
--replace-manifest'.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-170114543
**[Test build #2354 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2354/consoleFull)**
for PR 10256 at commit
[`6f3bbfd`](https://
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-170114213
**[Test build #2354 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2354/consoleFull)**
for PR 10256 at commit
[`6f3bbfd`](https://g
Github user BrianLondon commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-170111349
Rebased against current master, ran tests (except for docker) locally, and
tested kinesis example against aws hosted stream. All worked.
My preference woul
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-168074687
**[Test build #2271 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2271/consoleFull)**
for PR 10256 at commit
[`64858df`](https://
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-168073788
**[Test build #2271 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2271/consoleFull)**
for PR 10256 at commit
[`64858df`](https://g
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-166281133
Can this wait until 2.0? I think it's going to anyway now. Then the
dependency issue is fine.
---
If your project is set up for it, you can reply to this email and have
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-165907516
If the SQL/Hive tests are passing I'm good. @srowen usually has good
insight on dependency changes.
---
If your project is set up for it, you can reply to this email
Github user MLnick commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-165865257
I've also run the Kinesis tests locally (against a live AWS account)
against this PR and all tests pass. I'm +1 on this subject to agreement for the
jackson dependency b
Github user BrianLondon commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-164975298
Yeah, that looks like the same underlying issue of using AWS Java SDK
1.10.0 or later with Spark. I believe this version bump will fix it. You can
sometimes get a
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-164932897
FYI, a bunch of `spark-redshift` users have run into the Jackson dependency
conflict issue that's discussed here; see
https://github.com/databricks/spark-redshift/iss
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/10256#discussion_r47709922
--- Diff: pom.xml ---
@@ -169,7 +169,7 @@
${scala.version}
org.scala-lang
1.9.13
-2.4.4
+2.5.3
--- End diff --
Github user 3ourroom commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-164837398
NAVER - http://www.naver.com/
3ourr...@naver.com ëê» ë³´ë´ì ë©ì¼ ì´
ë¤ìê³¼ ê°ì ì´ì ë¡
Github user BrianLondon commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-164834977
I removed the explicit dependence on the AWS Java SDK. There's a newline
that was added to `KinesisReceiver.scala` since the last test run. I think if
tests are r
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/10256#discussion_r47591545
--- Diff: pom.xml ---
@@ -169,7 +169,7 @@
${scala.version}
org.scala-lang
1.9.13
-2.4.4
+2.5.3
--- End diff --
Github user BrianLondon commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-164553929
It appears kcl pulls in 1.10.20. I'm all for dropping the explicit
dependency. That probably wouldn't have prevented the regression in Spark
1.5.2 though. At any
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-164536120
I'm not really sure why - skimming the git logs it seems the aws-sdk-java
sdk dependency was added at the same time as the kcl dependency.
---
If your project is set u
Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/10256#discussion_r47545169
--- Diff: pom.xml ---
@@ -169,7 +169,7 @@
${scala.version}
org.scala-lang
1.9.13
-2.4.4
+2.5.3
--- End diff --
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/10256#discussion_r47543005
--- Diff: pom.xml ---
@@ -169,7 +169,7 @@
${scala.version}
org.scala-lang
1.9.13
-2.4.4
+2.5.3
--- End diff --
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-164527869
I think the next step is getting one of the jenkins admins (maybe @srowen
or @brkyvz ) since they are already on this PR to tell jenkins ok to test and
maybe see if @ma
Github user MLnick commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-164528959
@brkyvz @holdenk @tdas any particular reason we depend on `aws-sdk-java`
explicitly? The KCL brings in those dependencies (in fact from `1.5.0` KCL only
depends on the n
Github user MLnick commented on a diff in the pull request:
https://github.com/apache/spark/pull/10256#discussion_r47540953
--- Diff: pom.xml ---
@@ -169,7 +169,7 @@
${scala.version}
org.scala-lang
1.9.13
-2.4.4
+2.5.3
--- End diff --
Github user BrianLondon commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-164465363
@holdenk bump
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this fea
Github user BrianLondon commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-163985429
@brkyvz Pushed up the KCL version and added a newline to KinesisReciever.
Tests passed for me locally on a hadoop 2.6 build and the kinesis demos ran
against a liv
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-163945864
**[Test build #2203 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2203/consoleFull)**
for PR 10256 at commit
[`47d0a59`](https://
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-163927639
**[Test build #2203 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2203/consoleFull)**
for PR 10256 at commit
[`47d0a59`](https://g
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10256#discussion_r47350412
--- Diff: pom.xml ---
@@ -169,7 +169,7 @@
${scala.version}
org.scala-lang
1.9.13
-2.4.4
+2.5.3
--- End diff --
Github user brkyvz commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-163802104
@BrianLondon What happens if you change the version of KCL to 1.6.1? Could
you also please add a new line to anywhere in extras/kinesis-asl? Otherwise
Kinesis tests don'
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/10256#discussion_r47303246
--- Diff: pom.xml ---
@@ -169,7 +169,7 @@
${scala.version}
org.scala-lang
1.9.13
-2.4.4
+2.5.3
--- End diff --
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-163786007
/cc @brkyvz RE: the KCL library.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-163782601
Cool - if we don't need to upgrade the KCL library then thats probably
good. Maybe good to see if @tdas or someone similar can take a look?
---
If your project is set
Github user BrianLondon commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-163781633
I ran the demo with the following two commands against a stream
(test-stream1) on my AWS account. It performed as expected.
bin/run-example streaming.K
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-163778571
Note we also depend depend on com.fasterxml.jackson.core 2.4.4 in Spark
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-163778072
Have you tested the current kinesis and works with the new API? Looking
online it seems like we might also need to bump our kinesis client library if
we upgrade the gen
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10256#issuecomment-163748299
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your p
GitHub user BrianLondon opened a pull request:
https://github.com/apache/spark/pull/10256
[SPARK-12269][Streaming][Kinesis] Update aws-java-sdk version
The current Spark Streaming kinesis connector references a quite old
version 1.9.40 of the AWS Java SDK (1.10.40 is current). Numer
46 matches
Mail list logo
