[GitHub] spark issue #22192: [SPARK-24918][Core] Executor Plugin API

2018-09-05 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/22192 @vanzin @NiharS I am uncomfortable with that change as well - which is why I wanted the initialization to be pushed into a separate thread (and then join) - if we really need to set the context

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Rewrite condition when dedupli...

2018-09-05 Thread peter-toth
Github user peter-toth commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r215499599 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AttributeMap.scala --- @@ -23,12 +23,14 @@ package

[GitHub] spark issue #22337: [SPARK-25338][Test][kafka][kinesis][flume] Ensure to cal...

2018-09-05 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/22337 Sure, I just focused on files under `external`. Let me address other files, too. --- - To unsubscribe, e-mail:

[GitHub] spark issue #22344: [SPARK-25352][SQL] Perform ordered global limit when lim...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22344 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22344: [SPARK-25352][SQL] Perform ordered global limit when lim...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22344 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22344: [SPARK-25352][SQL] Perform ordered global limit when lim...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22344 **[Test build #95737 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95737/testReport)** for PR 22344 at commit

[GitHub] spark pull request #22344: [SPARK-25352][SQL] Perform ordered global limit w...

2018-09-05 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/22344 [SPARK-25352][SQL] Perform ordered global limit when limit number is bigger than topKSortFallbackThreshold ## What changes were proposed in this pull request? We have optimization on

[GitHub] spark issue #18142: [SPARK-20918] [SQL] Use FunctionIdentifier as function i...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18142 BTW, this is changed at Spark 2.3.0. How did we handle this before? --- - To unsubscribe, e-mail:

[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

2018-09-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/6 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22226: [SPARK-25252][SQL] Support arrays of any types by to_jso...

2018-09-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/6 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22340 It's great to have this in Spark 2.4! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22340 Thank you for merging, @srowen . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22337: [SPARK-25338][Test][kafka][kinesis][flume] Ensure to cal...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22337 Could you check the other suites like `ExternalAppendOnlyUnsafeRowArraySuite` and `TakeOrderedAndProjectSuite`, too? ---

[GitHub] spark pull request #22337: [SPARK-25338][Test][kafka][kinesis][flume] Ensure...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22337#discussion_r215491592 --- Diff: external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisStreamSuite.scala --- @@ -83,6 +83,7 @@ abstract class

[GitHub] spark pull request #22337: [SPARK-25338][Test][kafka][kinesis][flume] Ensure...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22337#discussion_r215491496 --- Diff: external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisInputDStreamBuilderSuite.scala --- @@ -41,6 +41,7 @@ class

[GitHub] spark pull request #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should ...

2018-09-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22340 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22337: [SPARK-25338][Test][kafka][kinesis][flume] Ensure...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22337#discussion_r215490599 --- Diff: external/flume/src/test/scala/org/apache/spark/streaming/flume/FlumePollingStreamSuite.scala --- @@ -58,6 +58,7 @@ class

[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...

2018-09-05 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22340 Merged to master (which looks like is still 2.4) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22337: [SPARK-25338][Test][kafka][kinesis][flume] Ensure to cal...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22337 Hi, @kiszk . Oh, did you check them all? Are these all of them? --- - To unsubscribe, e-mail:

[GitHub] spark issue #22339: SPARK-17159 Significant speed up for running spark strea...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22339 Hi, @ScrapCodes . Could you do the followings? - Update the title to `[SPARK-17159][SS]...` - Remove `Please review http://spark.apache.org/contributing.html ` from PR description

[GitHub] spark issue #22302: [SPARK-21786][SQL][FOLLOWUP] Add compressionCodec test f...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22302 **[Test build #95736 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95736/testReport)** for PR 22302 at commit

[GitHub] spark issue #18853: [SPARK-21646][SQL] Add new type coercion to compatible w...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18853 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95733/ Test FAILed. ---

[GitHub] spark issue #18853: [SPARK-21646][SQL] Add new type coercion to compatible w...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18853 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #18853: [SPARK-21646][SQL] Add new type coercion to compatible w...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18853 **[Test build #95733 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95733/testReport)** for PR 18853 at commit

[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22340 Thank you, @HyukjinKwon ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #22302: [SPARK-21786][SQL][FOLLOWUP] Add compressionCodec test f...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22302 Retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22302: [SPARK-21786][SQL][FOLLOWUP] Add compressionCodec test f...

2018-09-05 Thread fjh100456
Github user fjh100456 commented on the issue: https://github.com/apache/spark/pull/22302 The pr [#20120](https://github.com/apache/spark/pull/20120) was closed, instead [SPARK-23355](https://issues.apache.org/jira/browse/SPARK-23355) had resolved it. Does it mean the same thing

[GitHub] spark pull request #22270: [SPARK-25267][SQL][TEST] Disable ConvertToLocalRe...

2018-09-05 Thread dilipbiswal
Github user dilipbiswal commented on a diff in the pull request: https://github.com/apache/spark/pull/22270#discussion_r215485269 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -85,12 +85,12 @@ class DataFrameFunctionsSuite extends

[GitHub] spark issue #22270: [SPARK-25267][SQL][TEST] Disable ConvertToLocalRelation ...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22270 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22270: [SPARK-25267][SQL][TEST] Disable ConvertToLocalRelation ...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22270 **[Test build #95735 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95735/testReport)** for PR 22270 at commit

[GitHub] spark issue #22329: [SPARK-25328][PYTHON] Add an example for having two colu...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22329 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95734/ Test PASSed. ---

[GitHub] spark issue #22270: [SPARK-25267][SQL][TEST] Disable ConvertToLocalRelation ...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22270 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22329: [SPARK-25328][PYTHON] Add an example for having two colu...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22329 **[Test build #95734 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95734/testReport)** for PR 22329 at commit

[GitHub] spark issue #22329: [SPARK-25328][PYTHON] Add an example for having two colu...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22329 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22270: [SPARK-25267][SQL][TEST] Disable ConvertToLocalRe...

2018-09-05 Thread dilipbiswal
Github user dilipbiswal commented on a diff in the pull request: https://github.com/apache/spark/pull/22270#discussion_r215485064 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -85,12 +85,12 @@ class DataFrameFunctionsSuite extends

[GitHub] spark pull request #22270: [SPARK-25267][SQL][TEST] Disable ConvertToLocalRe...

2018-09-05 Thread dilipbiswal
Github user dilipbiswal commented on a diff in the pull request: https://github.com/apache/spark/pull/22270#discussion_r215484920 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -85,12 +85,12 @@ class DataFrameFunctionsSuite extends

[GitHub] spark issue #22140: [SPARK-25072][PySpark] Forbid extra value for custom Row

2018-09-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22140 @BryanCutler, for https://github.com/apache/spark/pull/22140#issuecomment-414802978, yea, to me it looks less sense actually but seems at least working for now: ```python from

[GitHub] spark issue #22271: [SPARK-25268][GraphX]run Parallel Personalized PageRank ...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22271 **[Test build #4332 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4332/testReport)** for PR 22271 at commit

[GitHub] spark pull request #22140: [SPARK-25072][PySpark] Forbid extra value for cus...

2018-09-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22140#discussion_r215483530 --- Diff: python/pyspark/sql/tests.py --- @@ -269,6 +269,10 @@ def test_struct_field_type_name(self): struct_field = StructField("a",

[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...

2018-09-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22340 Seems okay to me too --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #22287: [SPARK-25135][SQL] FileFormatWriter should respec...

2018-09-05 Thread wangyum
Github user wangyum closed the pull request at: https://github.com/apache/spark/pull/22287 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22320: [SPARK-25313][SQL]Fix regression in FileFormatWriter out...

2018-09-05 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22320 @gengliangwang We need backport this pr to branch-2.3. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22329: [SPARK-25328][PYTHON] Add an example for having two colu...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22329 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22329: [SPARK-25328][PYTHON] Add an example for having two colu...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22329 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22307: [SPARK-25301][SQL] When a view uses an UDF from a non de...

2018-09-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22307 @vinod, see the discussion made in https://github.com/apache/spark/pull/18142. Shall we close this? cc @cloud-fan as well. ---

[GitHub] spark issue #18142: [SPARK-20918] [SQL] Use FunctionIdentifier as function i...

2018-09-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18142 I agree with this change too for clarification. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22329: [SPARK-25328][PYTHON] Add an example for having two colu...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22329 **[Test build #95734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95734/testReport)** for PR 22329 at commit

[GitHub] spark issue #18142: [SPARK-20918] [SQL] Use FunctionIdentifier as function i...

2018-09-05 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18142 @HyukjinKwon Thanks for the note! I think this behavior is better, I'm adding a `release_note` tag to the JIRA ticket, so that we don't forget to mention it in release notes. ---

[GitHub] spark pull request #22320: [SPARK-25313][SQL]Fix regression in FileFormatWri...

2018-09-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22320 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22320: [SPARK-25313][SQL]Fix regression in FileFormatWriter out...

2018-09-05 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22320 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #22320: [SPARK-25313][SQL]Fix regression in FileFormatWri...

2018-09-05 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22320#discussion_r215479502 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala --- @@ -82,7 +83,7 @@ case class

[GitHub] spark issue #21968: [SPARK-24999][SQL]Reduce unnecessary 'new' memory operat...

2018-09-05 Thread heary-cao
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/21968 cc @cloud-fan @hvanhovell @maropu --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #21860: [SPARK-24901][SQL]Merge the codegen of RegularHashMap an...

2018-09-05 Thread heary-cao
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/21860 cc @cloud-fan @hvanhovell @maropu --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #22302: [SPARK-21786][SQL][FOLLOWUP] Add compressionCodec test f...

2018-09-05 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22302 btw, the behaivour differences of `TABLEPROPERTIES` and `OPTIONS` have already been documented somewhere? https://github.com/apache/spark/pull/20120#issuecomment-354704862 ---

[GitHub] spark issue #18142: [SPARK-20918] [SQL] Use FunctionIdentifier as function i...

2018-09-05 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18142 I am leaving a note (at least to myself) since it looked anyhow caused behaviour change. With this statements in Hive side: ```sql CREATE TABLE emp AS SELECT

[GitHub] spark issue #22302: [SPARK-21786][SQL][FOLLOWUP] Add compressionCodec test f...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22302 nit. Maybe, it seems that you removed `##`. You may want to simplify the PR description. This only re-enables the test cases of previous your PR. ``` ## What changes were proposed in

[GitHub] spark issue #22302: [SPARK-21786][SQL][FOLLOWUP] Add compressionCodec test f...

2018-09-05 Thread fjh100456
Github user fjh100456 commented on the issue: https://github.com/apache/spark/pull/22302 @dongjoon-hyun @maropu I'm so sorry, and I have changed the description. Is it ok now ? --- - To unsubscribe, e-mail:

[GitHub] spark issue #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3

2018-09-05 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22327 I think that is reasonable personally --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-09-05 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r215471255 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala --- @@ -60,7 +62,23 @@ private[spark] case

[GitHub] spark issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Struc...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22282 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-09-05 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r215471079 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -225,6 +225,60 @@ private[spark] object

[GitHub] spark issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Struc...

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22282 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95731/ Test FAILed. ---

[GitHub] spark issue #22282: [SPARK-23539][SS] Add support for Kafka headers in Struc...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22282 **[Test build #95731 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95731/testReport)** for PR 22282 at commit

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-09-05 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r215470727 --- Diff: docs/security.md --- @@ -722,6 +722,62 @@ with encryption, at least. The Kerberos login will be periodically renewed using the provided

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-09-05 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r215470379 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadoopsteps/HadoopStepsOrchestrator.scala --- @@ -0,0

[GitHub] spark issue #22327: [SPARK-25330][BUILD] Revert Hadoop 2.7 to 2.7.3

2018-09-05 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22327 How about revert it to branch-2.3 as we are going to release 2.3.2? We have time to fix it before releasing 2.4.0. --- - To

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-09-05 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r215470115 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadoopsteps/HadoopKerberosSecretResolverStep.scala ---

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-09-05 Thread ifilonenko
Github user ifilonenko commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r215469488 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/hadoopsteps/HadoopKerberosKeytabResolverStep.scala ---

[GitHub] spark issue #22192: [SPARK-24918][Core] Executor Plugin API

2018-09-05 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22192 > It seems like the line Thread.currentThread().setContextClassLoader(replClassLoader) is causing the pyspark failures What if you restore the original class loader after initializing the

[GitHub] spark issue #22192: [SPARK-24918][Core] Executor Plugin API

2018-09-05 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22192 > I'll change the config documentation to specify YARN only, hopefully it's not a huge issue. I don't think you're really understanding what I'm saying above. I'm not saying this feature

[GitHub] spark issue #22271: [SPARK-25268][GraphX]run Parallel Personalized PageRank ...

2018-09-05 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/22271 LGTM I tested this locally and confirmed it fixes the serialization issue. Thank you @shahidki31 ! Merging with master after fresh tests finish ---

[GitHub] spark issue #22192: [SPARK-24918][Core] Executor Plugin API

2018-09-05 Thread NiharS
Github user NiharS commented on the issue: https://github.com/apache/spark/pull/22192 I'll change the config documentation to specify YARN only, hopefully it's not a huge issue. It seems like the line `Thread.currentThread().setContextClassLoader(replClassLoader)` is causing

[GitHub] spark issue #22262: [SPARK-25175][SQL] Field resolution should fail if there...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22262 Thank you, @seancxmao . I'll review tonight again. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22302: [SPARK-21786][SQL][FOLLOWUP] Add compressionCodec test f...

2018-09-05 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22302 @fjh100456 please do not break the PR format, and can you clean up, too? --- - To unsubscribe, e-mail:

[GitHub] spark issue #20999: [WIP][SPARK-14922][SPARK-23866][SQL] Support partition f...

2018-09-05 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/20999 still WIP? btw, it seems currently credits can go to multiple developers; https://github.com/apache/spark/pull/22324#issuecomment-418227335 ---

[GitHub] spark issue #22302: [SPARK-21786][SQL][FOLLOWUP] Add compressionCodec test f...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22302 @fjh100456 . I'm @dongjoon-hyun . I think you put a wrong person in the the PR description. :) BTW, you don't need to mention the person. It would be enough to mention SPARK JIRA IDs.

[GitHub] spark pull request #22324: [SPARK-25237][SQL] Remove updateBytesReadWithFile...

2018-09-05 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22324#discussion_r215461695 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala --- @@ -473,6 +476,27 @@ class FileBasedDataSourceSuite extends

[GitHub] spark pull request #21525: [SPARK-24513][ML] Attribute support in UnaryTrans...

2018-09-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21525#discussion_r215461652 --- Diff: mllib/src/main/scala/org/apache/spark/ml/Transformer.scala --- @@ -116,10 +116,17 @@ abstract class UnaryTransformer[IN, OUT, T <:

[GitHub] spark issue #18853: [SPARK-21646][SQL] Add new type coercion to compatible w...

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18853 **[Test build #95733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95733/testReport)** for PR 18853 at commit

[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22340 Could you review this Scala-2.12 PR again, @cloud-fan and @gatorsmile ? --- - To unsubscribe, e-mail:

[GitHub] spark issue #20887: [SPARK-23774][SQL] `Cast` to CHAR/VARCHAR should truncat...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20887 Based on the review opinion, I'll close this PR and JIRA issue for now. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #20887: [SPARK-23774][SQL] `Cast` to CHAR/VARCHAR should ...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun closed the pull request at: https://github.com/apache/spark/pull/20887 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22270: [SPARK-25267][SQL][TEST] Disable ConvertToLocalRelation ...

2018-09-05 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22270 Thank you! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #22323: [SPARK-25262][K8S] Allow SPARK_LOCAL_DIRS to be t...

2018-09-05 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22323#discussion_r215456606 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStep.scala --- @@ -45,6 +47,10 @@

[GitHub] spark pull request #22323: [SPARK-25262][K8S] Allow SPARK_LOCAL_DIRS to be t...

2018-09-05 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22323#discussion_r215456338 --- Diff: docs/running-on-kubernetes.md --- @@ -215,6 +215,19 @@ spark.kubernetes.driver.volumes.persistentVolumeClaim.checkpointpvc.options.clai

[GitHub] spark pull request #22323: [SPARK-25262][K8S] Allow SPARK_LOCAL_DIRS to be t...

2018-09-05 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22323#discussion_r215456006 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStep.scala --- @@ -22,6 +22,7 @@ import

[GitHub] spark pull request #21669: [SPARK-23257][K8S] Kerberos Support for Spark on ...

2018-09-05 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/21669#discussion_r215455515 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala --- @@ -60,7 +62,23 @@ private[spark] case

[GitHub] spark issue #21525: [SPARK-24513][ML] Attribute support in UnaryTransformer

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21525 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21525: [SPARK-24513][ML] Attribute support in UnaryTransformer

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21525 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95732/ Test PASSed. ---

[GitHub] spark issue #21525: [SPARK-24513][ML] Attribute support in UnaryTransformer

2018-09-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21525 **[Test build #95732 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95732/testReport)** for PR 21525 at commit

[GitHub] spark pull request #22335: [SPARK-25091][SQL] reduce the storage memory in E...

2018-09-05 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/22335#discussion_r215452975 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala --- @@ -646,8 +646,17 @@ private[spark] class AppStatusListener( }

[GitHub] spark issue #22146: [SPARK-24434][K8S] pod template files

2018-09-05 Thread mccheah
Github user mccheah commented on the issue: https://github.com/apache/spark/pull/22146 I think we still need some consensus on the driver container API. Personally I think it's clearer to use a configuration option to specifically select the container that is the driver, but this is

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-09-05 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r215452489 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesDriverBuilder.scala --- @@ -81,9 +95,12 @@

[GitHub] spark issue #22270: [SPARK-25267][SQL][TEST] Disable ConvertToLocalRelation ...

2018-09-05 Thread dilipbiswal
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/22270 @gatorsmile Yeah.. i will push the changes tonight for you to take a look. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-09-05 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r215451113 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala --- @@ -59,5 +66,28 @@ private[spark] object

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-09-05 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r215450800 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala --- @@ -74,8 +74,16 @@ private[spark] object

[GitHub] spark issue #18080: [Spark-20771][SQL] Make weekofyear more intuitive

2018-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18080 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22146: [SPARK-24434][K8S] pod template files

2018-09-05 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/22146#discussion_r215450600 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala --- @@ -59,5 +66,28 @@ private[spark] object

[GitHub] spark issue #22333: [SPARK-25335][BUILD] Skip Zinc downloading if it's insta...

2018-09-05 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22333 Thank you, @srowen . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #22179: [SPARK-25258][SPARK-23131][SPARK-25176][BUILD] Up...

2018-09-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22179 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

  1   2   3   4   5   6   >