[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-06-20 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21588 Yup, it will still fail but it fixes everything else to make it working with Hadoop 3 within Spark. I think the current change is minimised as the current status as is and I meant to target

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21594 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92145/ Test PASSed. ---

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21594 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21594 **[Test build #92145 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92145/testReport)** for PR 21594 at commit

[GitHub] spark issue #21602: [SPARK-24613][SQL] Cache with UDF could not be matched w...

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21602 **[Test build #92149 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92149/testReport)** for PR 21602 at commit

[GitHub] spark issue #21602: [SPARK-24613][SQL] Cache with UDF could not be matched w...

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21602 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21602: [SPARK-24613][SQL] Cache with UDF could not be matched w...

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21602 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/364/

[GitHub] spark pull request #21602: [SPARK-24613][SQL] Cache with UDF could not be ma...

2018-06-20 Thread maryannxue
GitHub user maryannxue opened a pull request: https://github.com/apache/spark/pull/21602 [SPARK-24613][SQL] Cache with UDF could not be matched with subsequent dependent caches ## What changes were proposed in this pull request? Wrap the logical plan with a

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-06-20 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21588 I'm talking about the `VersionsSuite` stuff. I think it needs to be a more conscious decision about what happens. If, when build with Hadoop 3, Spark will not support older versions of

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-06-20 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21588 @vanzin, which tests do you mean? Two types of tests are skipped. One is by external Hive's limit which we can't control and the other one (two tests) looks by a JDK bug which I think we

[GitHub] spark issue #21601: [SPARK-24610] fix reading small files via wholeTextFiles

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21601 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92144/ Test PASSed. ---

[GitHub] spark issue #21601: [SPARK-24610] fix reading small files via wholeTextFiles

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21601 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21601: [SPARK-24610] fix reading small files via wholeTextFiles

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21601 **[Test build #92144 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92144/testReport)** for PR 21601 at commit

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21050 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21050 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/363/

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-06-20 Thread huaxingao
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/21050 @ueshin @kiszk Thanks for your comments. I fixed the problems. I am not sure if I should use ```$i++ ``` or ```$i ++``` in the for loop. It seems other people use ```$i ++```, so I also used

[GitHub] spark issue #21050: [SPARK-23912][SQL]add array_distinct

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21050 **[Test build #92148 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92148/testReport)** for PR 21050 at commit

[GitHub] spark issue #21462: [SPARK-24428][K8S] Fix unused code

2018-06-20 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/21462 @foxish gentle ping. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21598: [SPARK-24605][SQL] size(null) returns null instead of -1

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21598 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21598: [SPARK-24605][SQL] size(null) returns null instead of -1

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21598 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92142/ Test PASSed. ---

[GitHub] spark issue #21598: [SPARK-24605][SQL] size(null) returns null instead of -1

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21598 **[Test build #92142 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92142/testReport)** for PR 21598 at commit

[GitHub] spark issue #21577: [SPARK-24589][core] Correctly identify tasks in output c...

2018-06-20 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21577 the code here lgtm, I was trying to make one more pass through all the scenarios but got stuck in meetings, will try to do it later tonight or tomorrow morning but we can always have another

[GitHub] spark issue #21558: [SPARK-24552][SQL] Use task ID instead of attempt number...

2018-06-20 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21558 ping @rdblue ^ . If I don't hear tomorrow, will file separate jira. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #21593: [SPARK-24578][Core] Cap sub-region's size of retu...

2018-06-20 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21593 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21599: [SPARK-24598][SQL] Overflow on arithmetic operations ret...

2018-06-20 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21599 @cloud-fan @gatorsmile the main issue which is causing the UT failures, now, is that since before we were allowing overflows, in aggregations we could have an overflow eventually fixed by another

[GitHub] spark issue #21593: [SPARK-24578][Core] Cap sub-region's size of returned ni...

2018-06-20 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21593 Thanks! Merging to master and 2.3. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #21593: [SPARK-24578][Core] Cap sub-region's size of retu...

2018-06-20 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/21593#discussion_r196947295 --- Diff: common/network-common/src/main/java/org/apache/spark/network/protocol/MessageWithHeader.java --- @@ -137,30 +137,15 @@ protected void

[GitHub] spark issue #21073: [SPARK-23936][SQL] Implement map_concat

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21073 **[Test build #92147 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92147/testReport)** for PR 21073 at commit

[GitHub] spark pull request #21593: [SPARK-24578][Core] Cap sub-region's size of retu...

2018-06-20 Thread WenboZhao
Github user WenboZhao commented on a diff in the pull request: https://github.com/apache/spark/pull/21593#discussion_r196939992 --- Diff: common/network-common/src/main/java/org/apache/spark/network/protocol/MessageWithHeader.java --- @@ -137,30 +137,15 @@ protected void

[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21581 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21581 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92141/ Test PASSed. ---

[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21581 **[Test build #92141 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92141/testReport)** for PR 21581 at commit

[GitHub] spark pull request #21593: [SPARK-24578][Core] Cap sub-region's size of retu...

2018-06-20 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/21593#discussion_r196933769 --- Diff: common/network-common/src/main/java/org/apache/spark/network/protocol/MessageWithHeader.java --- @@ -137,30 +137,15 @@ protected void

[GitHub] spark issue #21560: [SPARK-24386][SS] coalesce(1) aggregates in continuous p...

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21560 **[Test build #92146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92146/testReport)** for PR 21560 at commit

[GitHub] spark pull request #21560: [SPARK-24386][SS] coalesce(1) aggregates in conti...

2018-06-20 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21560#discussion_r196930368 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousDataSourceRDD.scala --- @@ -51,7 +51,7 @@ class

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-20 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21596 Which is, btw, a way of saying you should run `dev/run-tests` locally, at least, when changing a dependency. --- - To

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-20 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21596 I agree we should upgrade, but just changing the pom version will not work. I have an internal patch for this and I needed small changes in a few other places... ---

[GitHub] spark issue #21595: [MINOR][SQL] Remove invalid comment from SparkStrategies

2018-06-20 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21595 @HyukjinKwon @hvanhovell Thanks for reviewing and merging! --- - To unsubscribe, e-mail:

[GitHub] spark pull request #21560: [SPARK-24386][SS] coalesce(1) aggregates in conti...

2018-06-20 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21560#discussion_r196924994 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousCoalesceRDD.scala --- @@ -0,0 +1,93 @@ +/* + *

[GitHub] spark pull request #21575: [SPARK-24566][CORE] spark.storage.blockManagerSla...

2018-06-20 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/21575#discussion_r196924766 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -75,16 +76,18 @@ private[spark] class HeartbeatReceiver(sc: SparkContext,

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21594 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/362/

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21594 **[Test build #92145 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92145/testReport)** for PR 21594 at commit

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21594 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21560: [SPARK-24386][SS] coalesce(1) aggregates in conti...

2018-06-20 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/21560#discussion_r196921230 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala --- @@ -350,7 +350,14 @@ object

[GitHub] spark pull request #21593: [SPARK-24578][Core] Cap sub-region's size of retu...

2018-06-20 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/21593#discussion_r196918296 --- Diff: common/network-common/src/main/java/org/apache/spark/network/protocol/MessageWithHeader.java --- @@ -137,30 +137,15 @@ protected void

[GitHub] spark issue #21542: [SPARK-24529][Build][test-maven] Add spotbugs into maven...

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21542 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92139/ Test FAILed. ---

[GitHub] spark issue #21542: [SPARK-24529][Build][test-maven] Add spotbugs into maven...

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21542 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21542: [SPARK-24529][Build][test-maven] Add spotbugs into maven...

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21542 **[Test build #92139 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92139/testReport)** for PR 21542 at commit

[GitHub] spark issue #21577: [SPARK-24589][core] Correctly identify tasks in output c...

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21577 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92138/ Test PASSed. ---

[GitHub] spark issue #21577: [SPARK-24589][core] Correctly identify tasks in output c...

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21577 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21577: [SPARK-24589][core] Correctly identify tasks in output c...

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21577 **[Test build #92138 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92138/testReport)** for PR 21577 at commit

[GitHub] spark issue #21601: [SPARK-24610] fix reading small files via wholeTextFiles

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21601 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21601: [SPARK-24610] fix reading small files via wholeTextFiles

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21601 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/361/

[GitHub] spark issue #21601: [SPARK-24610] fix reading small files via wholeTextFiles

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21601 **[Test build #92144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92144/testReport)** for PR 21601 at commit

[GitHub] spark issue #21599: [SPARK-24598][SQL] Overflow on arithmetic operations ret...

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21599 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21599: [SPARK-24598][SQL] Overflow on arithmetic operations ret...

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21599 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92140/ Test FAILed. ---

[GitHub] spark issue #21599: [SPARK-24598][SQL] Overflow on arithmetic operations ret...

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21599 **[Test build #92140 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92140/testReport)** for PR 21599 at commit

[GitHub] spark issue #21601: [SPARK-24610] fix reading small files via wholeTextFiles

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21601 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21601: [SPARK-24610] fix reading small files via wholeTextFiles

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21601 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92143/ Test FAILed. ---

[GitHub] spark issue #21601: [SPARK-24610] fix reading small files via wholeTextFiles

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21601 **[Test build #92143 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92143/testReport)** for PR 21601 at commit

[GitHub] spark issue #21601: [SPARK-24610] fix reading small files via wholeTextFiles

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21601 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/360/

[GitHub] spark issue #21601: [SPARK-24610] fix reading small files via wholeTextFiles

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21601 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21601: [SPARK-24610] fix reading small files via wholeTextFiles

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21601 **[Test build #92143 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92143/testReport)** for PR 21601 at commit

[GitHub] spark pull request #21601: [SPARK-24610] fix reading small files via wholeTe...

2018-06-20 Thread dhruve
GitHub user dhruve opened a pull request: https://github.com/apache/spark/pull/21601 [SPARK-24610] fix reading small files via wholeTextFiles ## What changes were proposed in this pull request? The `WholeTextFileInputFormat` determines the `maxSplitSize` for the file/s being

[GitHub] spark issue #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidation

2018-06-20 Thread maryannxue
Github user maryannxue commented on the issue: https://github.com/apache/spark/pull/21594 @TomaszGaweda @viirya Nice suggestion about the doc. I'll update it. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #21594: [SPARK-24596][SQL] Non-cascading Cache Invalidati...

2018-06-20 Thread maryannxue
Github user maryannxue commented on a diff in the pull request: https://github.com/apache/spark/pull/21594#discussion_r196899266 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala --- @@ -107,22 +107,35 @@ class CacheManager extends Logging {

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21527 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21527 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92137/ Test PASSed. ---

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-20 Thread dilipbiswal
Github user dilipbiswal commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r196897983 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -65,13 +65,38 @@ class JDBCOptions(

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21527 **[Test build #92137 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92137/testReport)** for PR 21527 at commit

[GitHub] spark issue #21596: [SPARK-24601] Bump Jackson version

2018-06-20 Thread Fokko
Github user Fokko commented on the issue: https://github.com/apache/spark/pull/21596 This was more than a year ago, we should eventually upgrade.. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-20 Thread dilipbiswal
Github user dilipbiswal commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r196895138 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -65,13 +65,38 @@ class JDBCOptions(

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-06-20 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/21588 If jenkins is happy, this is good. * Be interesting to see what happens in a build with the hadoop-cloud-storage module, if it adds new dependencies * regarding commons-config, know

[GitHub] spark issue #21597: [SPARK-24603] Fix findTightestCommonType reference in co...

2018-06-20 Thread Fokko
Github user Fokko commented on the issue: https://github.com/apache/spark/pull/21597 @HyukjinKwon Ok, some Apache projects are really strict with the Jira tickets. @maropu I've updated the commit --- - To

[GitHub] spark issue #21598: [SPARK-24605][SQL] size(null) returns null instead of -1

2018-06-20 Thread MaxGekk
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21598 > I think the JIRA and this PR claim the current behaviour is righter? @HyukjinKwon Yes but changing current behavior can potentially break existing user's applications. I am not sure we can do

[GitHub] spark issue #21577: [SPARK-24589][core] Correctly identify tasks in output c...

2018-06-20 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21577 I pushed the change for that in: https://github.com/vanzin/spark/commit/e6a862ecb83c64a0ea2f5bd469bc0febe25e15ba In case anyone wants to take a look. ---

[GitHub] spark issue #21598: [SPARK-24605][SQL] size(null) returns null instead of -1

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21598 **[Test build #92142 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92142/testReport)** for PR 21598 at commit

[GitHub] spark pull request #21598: [SPARK-24605][SQL] size(null) returns null instea...

2018-06-20 Thread MaxGekk
Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/21598#discussion_r196891392 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -1314,6 +1314,13 @@ object SQLConf { "Other column

[GitHub] spark issue #21577: [SPARK-24589][core] Correctly identify tasks in output c...

2018-06-20 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21577 I filed SPARK-24611 to track some enhancements to this part of the code that have been discussed here. Of those, I'd consider the "use task IDs instead of TaskIdentifier" as something we could

[GitHub] spark pull request #21598: [SPARK-24605][SQL] size(null) returns null instea...

2018-06-20 Thread MaxGekk
Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/21598#discussion_r196888409 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -1314,6 +1314,13 @@ object SQLConf { "Other column

[GitHub] spark issue #21577: [SPARK-24589][core] Correctly identify tasks in output c...

2018-06-20 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21577 Sounds good to me (although I'm trying the change locally and unit tests are so far happy). --- - To unsubscribe, e-mail:

[GitHub] spark issue #21577: [SPARK-24589][core] Correctly identify tasks in output c...

2018-06-20 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21577 > Ah, right, d'oh. I just checked about whether stages register with the coordinator, and saw the duplicate registration for the resubmitted map stage. Yeah I noticed that to but I think

[GitHub] spark issue #21598: [SPARK-24605][SQL] size(null) returns null instead of -1

2018-06-20 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21598 re: https://github.com/apache/spark/pull/21598#issuecomment-398737499 I missed the default value. Shall we set it to `false`? I think the JIRA and this PR claim the current behaviour is

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21588 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21588 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92134/ Test PASSed. ---

[GitHub] spark pull request #21495: [SPARK-24418][Build] Upgrade Scala to 2.11.12 and...

2018-06-20 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21495#discussion_r196878408 --- Diff: repl/scala-2.11/src/main/scala/org/apache/spark/repl/SparkILoopInterpreter.scala --- @@ -21,8 +21,22 @@ import scala.collection.mutable

[GitHub] spark issue #21577: [SPARK-24589][core] Correctly identify tasks in output c...

2018-06-20 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/21577 > * t2 finishes before that kill message arrives, is allowed to commit. > If that can happen it would generate a duplicate map output; but my guess (hope?) is that the map output tracker would

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21588 **[Test build #92134 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92134/testReport)** for PR 21588 at commit

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21588 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92135/ Test PASSed. ---

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21588 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21577: [SPARK-24589][core] Correctly identify tasks in output c...

2018-06-20 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21577 > if its a map stage then I don't expect it to be asking to commit. Ah, right, d'oh. I just checked about whether stages register with the coordinator, and saw the duplicate registration for

[GitHub] spark issue #21588: [SPARK-24590][BUILD] Make Jenkins tests passed with hado...

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21588 **[Test build #92135 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92135/testReport)** for PR 21588 at commit

[GitHub] spark issue #21577: [SPARK-24589][core] Correctly identify tasks in output c...

2018-06-20 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21577 > The test I added can sort of illustrate that if you look at what happens. There are two stages (map stage 2, result stage 3), and the fetch failure causes a retry of stage 3 plus a resubmission

[GitHub] spark issue #21577: [SPARK-24589][core] Correctly identify tasks in output c...

2018-06-20 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21577 I was referring to a race caused by asynchronously killing speculative tasks. Granted it's incredibly unlikely to occur in real life: - in s1a1 1, t1 and t2 are started for the same

[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...

2018-06-20 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21581 LGTM pending tests. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21517: Testing k8s change - please ignore (13)

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21517 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/359/ ---

[GitHub] spark issue #21517: Testing k8s change - please ignore (13)

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/359/

[GitHub] spark issue #21517: Testing k8s change - please ignore (13)

2018-06-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21517 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...

2018-06-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21581 **[Test build #92141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92141/testReport)** for PR 21581 at commit

[GitHub] spark issue #21581: [SPARK-24574][SQL] array_contains, array_position, array...

2018-06-20 Thread ueshin
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/21581 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

<    1   2   3   4   5   >