[GitHub] spark pull request #15172: [SPARK-13331] AES support for over-the-wire encry...

2016-10-31 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/15172#discussion_r85848236 --- Diff: common/network-common/src/main/java/org/apache/spark/network/sasl/aes/AesCipher.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #15172: [SPARK-13331] AES support for over-the-wire encry...

2016-10-31 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/15172#discussion_r85847230 --- Diff: common/network-common/src/main/java/org/apache/spark/network/sasl/aes/AesCipher.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #15695: [SPARK-18143][SQL]Ignore Structured Streaming event logs...

2016-10-31 Thread zsxwing
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15695 Sure. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark pull request #15172: [SPARK-13331] AES support for over-the-wire encry...

2016-10-31 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/15172#discussion_r85849098 --- Diff: common/network-common/src/main/java/org/apache/spark/network/sasl/aes/AesCipher.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #15172: [SPARK-13331] AES support for over-the-wire encry...

2016-10-31 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/15172#discussion_r85847599 --- Diff: common/network-common/src/main/java/org/apache/spark/network/sasl/aes/AesCipher.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #15675: [SPARK-18144][SQL] logging StreamingQueryListener$QueryS...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15675 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67833/ Test PASSed. ---

[GitHub] spark pull request #15172: [SPARK-13331] AES support for over-the-wire encry...

2016-10-31 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/15172#discussion_r85850400 --- Diff: common/network-common/src/main/java/org/apache/spark/network/util/ByteArrayReadableChannel.java --- @@ -0,0 +1,71 @@ +/* + * Licensed to

[GitHub] spark pull request #15172: [SPARK-13331] AES support for over-the-wire encry...

2016-10-31 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/15172#discussion_r85849036 --- Diff: common/network-common/src/main/java/org/apache/spark/network/sasl/aes/AesCipher.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #15172: [SPARK-13331] AES support for over-the-wire encry...

2016-10-31 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/15172#discussion_r85849245 --- Diff: common/network-common/src/main/java/org/apache/spark/network/sasl/aes/AesCipher.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #15172: [SPARK-13331] AES support for over-the-wire encry...

2016-10-31 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/15172#discussion_r85851023 --- Diff: docs/configuration.md --- @@ -1435,6 +1435,21 @@ Apart from these, the following properties are also available, and may be useful

[GitHub] spark pull request #15172: [SPARK-13331] AES support for over-the-wire encry...

2016-10-31 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/15172#discussion_r85845790 --- Diff: common/network-common/src/main/java/org/apache/spark/network/sasl/SaslClientBootstrap.java --- @@ -88,9 +90,25 @@ public void

[GitHub] spark pull request #15172: [SPARK-13331] AES support for over-the-wire encry...

2016-10-31 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/15172#discussion_r85850499 --- Diff: common/network-common/src/main/java/org/apache/spark/network/sasl/aes/AesCipher.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #15172: [SPARK-13331] AES support for over-the-wire encry...

2016-10-31 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/15172#discussion_r85848147 --- Diff: common/network-common/src/main/java/org/apache/spark/network/sasl/aes/AesCipher.java --- @@ -0,0 +1,311 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request #15695: [SPARK-18143][SQL]Ignore Structured Streaming eve...

2016-10-31 Thread zsxwing
Github user zsxwing closed the pull request at: https://github.com/apache/spark/pull/15695 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #15172: [SPARK-13331] AES support for over-the-wire encry...

2016-10-31 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/15172#discussion_r85849596 --- Diff: common/network-common/src/main/java/org/apache/spark/network/sasl/aes/AesConfigMessage.java --- @@ -0,0 +1,104 @@ +/* + * Licensed to the

[GitHub] spark pull request #15172: [SPARK-13331] AES support for over-the-wire encry...

2016-10-31 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/15172#discussion_r85849759 --- Diff: common/network-common/src/main/java/org/apache/spark/network/sasl/aes/AesConfigMessage.java --- @@ -0,0 +1,104 @@ +/* + * Licensed to the

[GitHub] spark issue #15699: [SPARK-18030][Tests]Fix flaky FileStreamSourceSuite by n...

2016-10-31 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15699 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #15701: [SPARK-18167] [SQL] Also log all partitions when the SQL...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15701 **[Test build #67836 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67836/consoleFull)** for PR 15701 at commit

[GitHub] spark issue #15696: [SPARK-18024][SQL] Introduce an internal commit protocol...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15696 **[Test build #67840 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67840/consoleFull)** for PR 15696 at commit

[GitHub] spark issue #15701: [SPARK-18167] [SQL] Also log all partitions when the SQL...

2016-10-31 Thread yhuai
Github user yhuai commented on the issue: https://github.com/apache/spark/pull/15701 merging to master. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15677: [SPARK-17963][SQL][Documentation] Add examples (extend) ...

2016-10-31 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15677 @HyukjinKwon Sure, I can try it. Actually, we might need to find multiple typical function APIs as examples to discuss what are the best way to document/specify the argument/parameter

[GitHub] spark issue #15705: [SPARK-18183] [SPARK-18184] Fix INSERT [INTO|OVERWRITE] ...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15705 **[Test build #67846 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67846/consoleFull)** for PR 15705 at commit

[GitHub] spark pull request #15651: [SPARK-17972][SQL] Add Dataset.checkpoint() to tr...

2016-10-31 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15651 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #15666: [SPARK-11421] [Core][Python][R] Added ability for...

2016-10-31 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/15666#discussion_r85830311 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1700,19 +1700,34 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark issue #15699: [SPARK-18030][Tests]Fix flaky FileStreamSourceSuite by n...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15699 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15699: [SPARK-18030][Tests]Fix flaky FileStreamSourceSuite by n...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15699 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67830/ Test PASSed. ---

[GitHub] spark issue #15696: [SPARK-18024][SQL] Introduce an internal commit protocol...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15696 **[Test build #67840 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67840/consoleFull)** for PR 15696 at commit

[GitHub] spark pull request #15703: [SPARK-18186] Migrate HiveUDAFFunction to TypedIm...

2016-10-31 Thread liancheng
GitHub user liancheng opened a pull request: https://github.com/apache/spark/pull/15703 [SPARK-18186] Migrate HiveUDAFFunction to TypedImperativeAggregate for partial aggregation support ## What changes were proposed in this pull request? This PR migrates

[GitHub] spark issue #14444: [SPARK-16839] [SQL] redundant aliases after cleanupAlias...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/1 **[Test build #67837 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67837/consoleFull)** for PR 1 at commit

[GitHub] spark issue #14444: [SPARK-16839] [SQL] redundant aliases after cleanupAlias...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/1 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67837/ Test FAILed. ---

[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15688 **[Test build #67847 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67847/consoleFull)** for PR 15688 at commit

[GitHub] spark issue #13575: [SPARK-15472][SQL] Add support for writing in `csv`, `js...

2016-10-31 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/13575 @lw-lin sorry we haven't visited this pr in the past, but the goal of this pr will be accomplished by https://issues.apache.org/jira/browse/SPARK-17924 (as a side effect). It reduces a lot of

[GitHub] spark issue #15706: [SPARK-18189] [SQL] Fix serialization issue in KeyValueG...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15706 **[Test build #3385 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3385/consoleFull)** for PR 15706 at commit

[GitHub] spark issue #14444: [SPARK-16839] [SQL] redundant aliases after cleanupAlias...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/1 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67869/ Test FAILed. ---

[GitHub] spark issue #15709: [SPARK-18190][Build][SparkR] Fix R version to not the la...

2016-10-31 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15709 I see. That seems nice too but I hope the version is fixed if this one is also fine too.. I am worried if the version is suddenly bumped up and some tests are failed (at least, I was a

[GitHub] spark issue #14444: [SPARK-16839] [SQL] redundant aliases after cleanupAlias...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/1 **[Test build #67869 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67869/consoleFull)** for PR 1 at commit

[GitHub] spark issue #15693: [SPARK-18125][SQL] Fix a compilation error in codegen du...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15693 **[Test build #67850 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67850/consoleFull)** for PR 15693 at commit

[GitHub] spark issue #15675: [SPARK-18144][SQL] logging StreamingQueryListener$QueryS...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15675 **[Test build #67851 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67851/consoleFull)** for PR 15675 at commit

[GitHub] spark pull request #15696: [SPARK-18024][SQL] Introduce an internal commit p...

2016-10-31 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15696#discussion_r85860514 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileCommitProtocol.scala --- @@ -0,0 +1,223 @@ +/* + * Licensed to the

[GitHub] spark pull request #15696: [SPARK-18024][SQL] Introduce an internal commit p...

2016-10-31 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15696#discussion_r85860719 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriteOutput.scala --- @@ -218,22 +210,24 @@ object WriteOutput extends Logging {

[GitHub] spark pull request #15696: [SPARK-18024][SQL] Introduce an internal commit p...

2016-10-31 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15696#discussion_r85859487 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileCommitProtocol.scala --- @@ -0,0 +1,223 @@ +/* + * Licensed to the

[GitHub] spark pull request #15696: [SPARK-18024][SQL] Introduce an internal commit p...

2016-10-31 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15696#discussion_r85859041 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileCommitProtocol.scala --- @@ -0,0 +1,223 @@ +/* + * Licensed to the

[GitHub] spark pull request #15696: [SPARK-18024][SQL] Introduce an internal commit p...

2016-10-31 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15696#discussion_r85859292 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileCommitProtocol.scala --- @@ -0,0 +1,223 @@ +/* + * Licensed to the

[GitHub] spark pull request #15696: [SPARK-18024][SQL] Introduce an internal commit p...

2016-10-31 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15696#discussion_r85859055 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileCommitProtocol.scala --- @@ -0,0 +1,223 @@ +/* + * Licensed to the

[GitHub] spark issue #15706: [SPARK-18189] [Core] Fix serialization issue in KeyValue...

2016-10-31 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/15706 Ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #15593: [SPARK-18060][ML] Avoid unnecessary computation for MLOR

2016-10-31 Thread sethah
Github user sethah commented on the issue: https://github.com/apache/spark/pull/15593 ping @MLnick @jkbradley This should be a nice performance boost for MLOR in ML, hopefully we can get it in for 2.1. If you get some time to review or run tests I'd really appreciate it. --- If

[GitHub] spark issue #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION should sup...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15704 **[Test build #67857 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67857/consoleFull)** for PR 15704 at commit

[GitHub] spark issue #15697: [SparkR][Test]:remove unnecessary suppressWarnings

2016-10-31 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15697 Build started: [SparkR] `ALL`

[GitHub] spark issue #15677: [WIP][SPARK-17963][SQL][Documentation] Add examples (ext...

2016-10-31 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15677 Thank you for your work! I might offend you in my previous comments. That's certainly not my intention. Previously, I worked for the infrastructure software on mainframe. IMO, Spark is

[GitHub] spark issue #15667: [SPARK-18107][SQL] Insert overwrite statement runs much ...

2016-10-31 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/15667 Let's do it in a follow-up. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #15696: [SPARK-18024][SQL] Introduce an internal commit protocol...

2016-10-31 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15696 Closing this in favor of https://github.com/apache/spark/pull/15707 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #15696: [SPARK-18024][SQL] Introduce an internal commit p...

2016-10-31 Thread rxin
Github user rxin closed the pull request at: https://github.com/apache/spark/pull/15696 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #15694: [SPARK-18179][SQL] Throws analysis exception with a prop...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15694 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15707: [SPARK-18024][SQL] Introduce an internal commit protocol...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15707 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15707: [SPARK-18024][SQL] Introduce an internal commit protocol...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15707 **[Test build #67855 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67855/consoleFull)** for PR 15707 at commit

[GitHub] spark issue #15694: [SPARK-18179][SQL] Throws analysis exception with a prop...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15694 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67856/ Test PASSed. ---

[GitHub] spark issue #15673: [SPARK-17992][SQL] Return all partitions from HiveShim w...

2016-10-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15673 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #15707: [SPARK-18024][SQL] Introduce an internal commit protocol...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15707 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67855/ Test PASSed. ---

[GitHub] spark issue #15707: [SPARK-18024][SQL] Introduce an internal commit protocol...

2016-10-31 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/15707 This LGTM, just a minor comment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #15669: [SPARK-18160][CORE][YARN] spark.files should not ...

2016-10-31 Thread zjffdu
Github user zjffdu commented on a diff in the pull request: https://github.com/apache/spark/pull/15669#discussion_r85872343 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -1716,29 +1716,12 @@ class SparkContext(config: SparkConf) extends Logging {

[GitHub] spark issue #15669: [SPARK-18160][CORE][YARN] spark.files should not be pass...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15669 **[Test build #67868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67868/consoleFull)** for PR 15669 at commit

[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...

2016-10-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15688 @gatorsmile that's a good point. I checked with hive, the behaviour is: 1. if the given partition spec specifies all partition columns, throw exception if the partition not exist 2.

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2016-10-31 Thread zjffdu
Github user zjffdu commented on the issue: https://github.com/apache/spark/pull/13599 Thanks for the review @mridulm , this approach is trying the move the overhead from user to cluster. User just need to specify the requirement file and spark will set up the virtualenv

[GitHub] spark issue #15710: [SPARK-18025] Use commit protocol API in structured stre...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15710 **[Test build #67873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67873/consoleFull)** for PR 15710 at commit

[GitHub] spark issue #15302: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION should sup...

2016-10-31 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15302 Thank you, @hvanhovell ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #15414: [SPARK-17848][ML] Move LabelCol datatype cast into Predi...

2016-10-31 Thread sethah
Github user sethah commented on the issue: https://github.com/apache/spark/pull/15414 LGTM as well after adding @jkbradley's suggestion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION should sup...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15704 **[Test build #67843 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67843/consoleFull)** for PR 15704 at commit

[GitHub] spark issue #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION should sup...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15704 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67843/ Test FAILed. ---

[GitHub] spark issue #15704: [SPARK-17732][SQL] ALTER TABLE DROP PARTITION should sup...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15704 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15706: [SPARK-18189] [SQL] Fix serialization issue in KeyValueG...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15706 **[Test build #3383 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3383/consoleFull)** for PR 15706 at commit

[GitHub] spark issue #15706: [SPARK-18189] [SQL] Fix serialization issue in KeyValueG...

2016-10-31 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15706 LGTM pending Jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark pull request #15707: [SPARK-18024][SQL] Introduce an internal commit p...

2016-10-31 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/15707 [SPARK-18024][SQL] Introduce an internal commit protocol API - rebased ## What changes were proposed in this pull request? This patch introduces an internal commit protocol API that is used by the

[GitHub] spark issue #15703: [SPARK-18186] Migrate HiveUDAFFunction to TypedImperativ...

2016-10-31 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15703 cc @tejasapatil --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #15707: [SPARK-18024][SQL] Introduce an internal commit protocol...

2016-10-31 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15707 This is the same as https://github.com/apache/spark/pull/15696 but rebased with https://github.com/apache/spark/pull/15633 --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #15667: [SPARK-18107][SQL] Insert overwrite statement runs much ...

2016-10-31 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/15667 lgtm if tests pass --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #15414: [SPARK-17848][ML] Move LabelCol datatype cast into Predi...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15414 **[Test build #67861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67861/consoleFull)** for PR 15414 at commit

[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15688 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67847/ Test PASSed. ---

[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15688 **[Test build #67847 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67847/consoleFull)** for PR 15688 at commit

[GitHub] spark issue #15705: [SPARK-18183] [SPARK-18184] Fix INSERT [INTO|OVERWRITE] ...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15705 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67848/ Test FAILed. ---

[GitHub] spark issue #15633: [SPARK-18087] [SQL] Optimize insert to not require REPAI...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15633 **[Test build #3381 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3381/consoleFull)** for PR 15633 at commit

[GitHub] spark issue #15705: [SPARK-18183] [SPARK-18184] Fix INSERT [INTO|OVERWRITE] ...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15705 **[Test build #67848 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67848/consoleFull)** for PR 15705 at commit

[GitHub] spark issue #15688: [SPARK-18173][SQL] data source tables should support tru...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15688 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15707: [SPARK-18024][SQL] Introduce an internal commit protocol...

2016-10-31 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/15707 This lgtm, modulo the comments in #15696 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15706: [SPARK-18189] [SQL] Fix serialization issue in KeyValueG...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15706 **[Test build #3383 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3383/consoleFull)** for PR 15706 at commit

[GitHub] spark issue #15705: [SPARK-18183] [SPARK-18184] Fix INSERT [INTO|OVERWRITE] ...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15705 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15675: [SPARK-18144][SQL] logging StreamingQueryListener$QueryS...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15675 **[Test build #67851 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67851/consoleFull)** for PR 15675 at commit

[GitHub] spark issue #15675: [SPARK-18144][SQL] logging StreamingQueryListener$QueryS...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15675 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67851/ Test PASSed. ---

[GitHub] spark issue #15666: [SPARK-11421] [Core][Python][R] Added ability for addJar...

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15666 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67860/ Test PASSed. ---

[GitHub] spark issue #15673: [SPARK-17992][SQL] Return all partitions from HiveShim w...

2016-10-31 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15673 @mallman can you bring this up-to-date? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #15172: [SPARK-13331] AES support for over-the-wire encry...

2016-10-31 Thread cjjnjust
Github user cjjnjust commented on a diff in the pull request: https://github.com/apache/spark/pull/15172#discussion_r85875690 --- Diff: common/network-common/src/main/java/org/apache/spark/network/sasl/SaslRpcHandler.java --- @@ -80,46 +84,71 @@ public void

[GitHub] spark pull request #15710: [SPARK-18025] Use commit protocol API in structur...

2016-10-31 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/15710 [SPARK-18025] Use commit protocol API in structured streaming ## What changes were proposed in this pull request? This patch adds a new commit protocol implementation ManifestFileCommitProtocol

[GitHub] spark issue #15709: [SPARK-18190][Build][SparkR] Fix R version to not the la...

2016-10-31 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/15709 This approach is fine - The other thing we could do is just use the latest stable version as described in https://cloud.r-project.org/bin/windows/base/ - If you see the link at the bottom it says

[GitHub] spark pull request #15706: [SPARK-18189] [Core] Fix serialization issue in K...

2016-10-31 Thread seyfe
GitHub user seyfe opened a pull request: https://github.com/apache/spark/pull/15706 [SPARK-18189] [Core] Fix serialization issue in KeyValueGroupedDataset ## What changes were proposed in this pull request? Likewise

[GitHub] spark issue #15696: [SPARK-18024][SQL] Introduce an internal commit protocol...

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15696 **[Test build #67852 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67852/consoleFull)** for PR 15696 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed-delay based Event Time Watermarks

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15702: [SPARK-18124] Observed-delay based Event Time Watermarks

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #67839 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67839/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15633: [SPARK-18087] [SQL] Optimize insert to not require REPAI...

2016-10-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15633 LGTM, pending jenkins --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #15702: [SPARK-18124] Observed-delay based Event Time Watermarks

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67839/ Test FAILed. ---

[GitHub] spark pull request #15673: [SPARK-17992][SQL] Return all partitions from Hiv...

2016-10-31 Thread ericl
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/15673#discussion_r85856641 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala --- @@ -585,7 +586,31 @@ private[client] class Shim_v0_13 extends Shim_v0_12

[GitHub] spark issue #15692: [SPARK-18177][ML][PYSPARK] Add missing 'subsamplingRate'...

2016-10-31 Thread zhengruifeng
Github user zhengruifeng commented on the issue: https://github.com/apache/spark/pull/15692 @jkbradley I add this param for GBTClassifier and GBTRegressor. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

<    1   2   3   4   5   6   >