[GitHub] spark pull request: [SPARK-12921] Use SparkHadoopUtil reflection i...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/10843#issuecomment-173122192
  
/cc @aarondav


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10837#issuecomment-173122146
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10837#issuecomment-173122147
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49763/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12921] Use SparkHadoopUtil reflection i...

[JoshRosen]

GitHub user JoshRosen opened a pull request:

https://github.com/apache/spark/pull/10843

[SPARK-12921] Use SparkHadoopUtil reflection in 
SpecificParquetRecordReaderBase

It looks like there's one place left in the codebase, 
SpecificParquetRecordReaderBase, where we didn't use SparkHadoopUtil's 
reflective accesses of TaskAttemptContext methods, which could create problems 
when using a single Spark artifact with both Hadoop 1.x and 2.x.

This patch fixes that using reflection. I'm not sure why our custom style 
checker rules didn't catch this case.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JoshRosen/spark SPARK-12921

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10843.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10843


commit 7cf0948b379af54d01329f8bfdebe2a603ea38b9
Author: Josh Rosen 
Date:   2016-01-20T07:48:12Z

Use SparkHadoopUtil.getConfigurationFromJobContext() in 
SpecificParquetRecordReaderBase




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP][SPARK-12828][SQL]add natural join suppor...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10762#issuecomment-173121947
  
**[Test build #49768 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49768/consoleFull)**
 for PR 10762 at commit 
[`25a7226`](https://github.com/apache/spark/commit/25a7226cb3a3039c704b6ca6f38e7a7b3cdb599e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10837#issuecomment-173121951
  
**[Test build #49763 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49763/consoleFull)**
 for PR 10837 at commit 
[`f5b3c18`](https://github.com/apache/spark/commit/f5b3c185971003be1459a06acab10dda9f1b9a31).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12699][SPARKR] R driver process should ...

[felixcheung]

Github user felixcheung commented on the pull request:

https://github.com/apache/spark/pull/10652#issuecomment-173121016
  
I realize that, my point is even in client mode the driver could be running 
on a worker machine, as in the case Spark job is submitted from another YARN 
app.







On Tue, Jan 19, 2016 at 11:22 PM -0800, "sun-rui" 
 wrote:





It is possible to get deploy mode from "spark.submit.deployMode", and check 
if it is "client". You can take a look at 
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/r/RUtils.scala#L49

---
Reply to this email directly or view it on GitHub:
https://github.com/apache/spark/pull/10652#issuecomment-173116720



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12848][SQL] Change parsed decimal liter...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10796#issuecomment-173120460
  
**[Test build #49767 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49767/consoleFull)**
 for PR 10796 at commit 
[`8bf2a27`](https://github.com/apache/spark/commit/8bf2a278915b9bb35424cfd91d2130018af973ed).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12881] [SQL] subexpress elimination in ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10814#issuecomment-173119258
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12881] [SQL] subexpress elimination in ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10814#issuecomment-173119260
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49762/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12881] [SQL] subexpress elimination in ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10814#issuecomment-173119061
  
**[Test build #49762 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49762/consoleFull)**
 for PR 10814 at commit 
[`d907949`](https://github.com/apache/spark/commit/d9079498e1c82cd1ed349213b63be78af1ab1bac).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10837#issuecomment-173118683
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10837#issuecomment-173118684
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49761/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10837#issuecomment-173118503
  
**[Test build #49761 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49761/consoleFull)**
 for PR 10837 at commit 
[`93a7728`](https://github.com/apache/spark/commit/93a7728b1cd5623feba4a17c39988c639e6a5b5d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12797] [SQL] Generated TungstenAggregat...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/10840#issuecomment-173117446
  
What's the difference between this one and #10786?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12616] [SQL] Making Logical Operator `U...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10577#issuecomment-173116859
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49760/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12616] [SQL] Making Logical Operator `U...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10577#issuecomment-173116857
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12616] [SQL] Making Logical Operator `U...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10577#issuecomment-173116680
  
**[Test build #49760 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49760/consoleFull)**
 for PR 10577 at commit 
[`c63f237`](https://github.com/apache/spark/commit/c63f237b6de61c700cec3544820674c1dcb94334).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12699][SPARKR] R driver process should ...

[sun-rui]

Github user sun-rui commented on the pull request:

https://github.com/apache/spark/pull/10652#issuecomment-173116720
  
It is possible to get deploy mode from "spark.submit.deployMode", and check 
if it is "client". You can take a look at 
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/api/r/RUtils.scala#L49


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12610][SQL][WIP] Add Anti join operator...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10563#issuecomment-173115403
  
**[Test build #49766 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49766/consoleFull)**
 for PR 10563 at commit 
[`70b7804`](https://github.com/apache/spark/commit/70b7804b45de44d0adfab74d1ff38651fdd5ff2e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12699][SPARKR] R driver process should ...

[felixcheung]

Github user felixcheung commented on the pull request:

https://github.com/apache/spark/pull/10652#issuecomment-173110574
  
I don't know if there is a way to distinguish that.
It could be `spark-submit` or calling `SparkSubmit` class from Oozie and 
running the job in YARN client mode in which case the driver is actually 
running on a worker, which could be the same worker running executors.

I guess we could explicitly bypass this if the cluster manager is `LOCAL`?




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12705] [SPARK-10777] [SQL] Analyzer Rul...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10678#issuecomment-173110456
  
**[Test build #49765 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49765/consoleFull)**
 for PR 10678 at commit 
[`1964884`](https://github.com/apache/spark/commit/1964884afee035f1e13bd248073656ec43de2223).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12689][SQL] Migrate DDL parsing to the ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10723#issuecomment-173109235
  
**[Test build #49764 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49764/consoleFull)**
 for PR 10723 at commit 
[`559083e`](https://github.com/apache/spark/commit/559083eddd0129edeba128927be5810e3a53dc9d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-12898. Consider having dummyCallSite for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10825#issuecomment-173109060
  
**[Test build #2419 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2419/consoleFull)**
 for PR 10825 at commit 
[`51c7aba`](https://github.com/apache/spark/commit/51c7aba3e831db5048f0b4185f7017456bd22428).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-12898. Consider having dummyCallSite for...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10825#issuecomment-173108923
  
**[Test build #2419 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2419/consoleFull)**
 for PR 10825 at commit 
[`51c7aba`](https://github.com/apache/spark/commit/51c7aba3e831db5048f0b4185f7017456bd22428).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-12898. Consider having dummyCallSite for...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/10825#issuecomment-173108734
  
LGTM pending tests.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/10836#discussion_r50219109
  
--- Diff: R/README.md ---
@@ -1,6 +1,16 @@
 # R on Spark
 
 SparkR is an R package that provides a light-weight frontend to use Spark 
from R.
+### Installing sparkR
+
+Libraries of sparkR need to be created in `$SPARK_HOME/R/lib`. This can be 
done by running the script `$SPARK_HOME/R/install-dev.sh`.
--- End diff --

actually, this line
```.libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), 
.libPaths()))```
added SPARK_HOME/R/lib into R's lib path and allows R, any running version, 
to load SparkR package from there - SparkR packages does not need to be 
installed with `R CMD INSTALL` (in install-dev.sh) at all.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12204][SPARKR] Implement drop method fo...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/10201#discussion_r50219056
  
--- Diff: docs/sql-programming-guide.md ---
@@ -2073,6 +2073,8 @@ options.
  --conf spark.sql.hive.thriftServer.singleSession=true \
  ...
{% endhighlight %}
+ - Since 1.6.1, withColumn method in sparkR supports adding a new column 
to or replacing existing columns
--- End diff --

maybe we want to put this in the R migration guide session instead of SQL? 
or both?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12689][SQL] Migrate DDL parsing to the ...

[viirya]

Github user viirya commented on the pull request:

https://github.com/apache/spark/pull/10723#issuecomment-173108390
  
Why jenkin continues throw exceptions like:

[info] Exception encountered when attempting to run a suite with class 
name: 
org.apache.spark.sql.execution.columnar.InMemoryColumnarQuerySuite *** 
ABORTED *** (70 milliseconds)
[info]   org.apache.spark.SparkException: Only one SparkContext may be 
running in this JVM
(see SPARK-2243). To ignore this error, set 
spark.driver.allowMultipleContexts = true. The currently
running SparkContext was created at:
[info] 
org.apache.spark.sql.sources.TableScanSuite.beforeAll(TableScanSuite.scala:129)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12689][SQL] Migrate DDL parsing to the ...

[viirya]

Github user viirya commented on the pull request:

https://github.com/apache/spark/pull/10723#issuecomment-173108412
  
retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12224][SPARKR] R support for JDBC sourc...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/10480#discussion_r50218806
  
--- Diff: R/pkg/R/DataFrame.R ---
@@ -2401,4 +2401,41 @@ setMethod("str",
 cat(paste0("\nDisplaying first ", ncol(localDF), " columns 
only."))
   }
 }
-  })
\ No newline at end of file
+  })
+
+#' Saves the content of the DataFrame to an external database table via 
JDBC
+#'
+#' Additional JDBC database connection properties can be set (...)
+#'
+#' Also, mode is used to specify the behavior of the save operation when
+#' data already exists in the data source. There are four modes: \cr
--- End diff --

yes, it forces a new line in the generated doc, otherwise roxygen2 removes 
new line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-12417. [SQL] Orc bloom filter options ar...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10842#issuecomment-173107729
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10837#issuecomment-173107401
  
**[Test build #49763 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49763/consoleFull)**
 for PR 10837 at commit 
[`f5b3c18`](https://github.com/apache/spark/commit/f5b3c185971003be1459a06acab10dda9f1b9a31).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10263] [ML] Add @Since annotation to ml...

[taishi-oss]

Github user taishi-oss commented on the pull request:

https://github.com/apache/spark/pull/8935#issuecomment-173107263
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-12417. [SQL] Orc bloom filter options ar...

[rajeshbalamohan]

GitHub user rajeshbalamohan opened a pull request:

https://github.com/apache/spark/pull/10842

SPARK-12417. [SQL] Orc bloom filter options are not propagated during…

Add option to have bloom filters in ORC write codepath.  Added changes to 
apply cleanly in master branch.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajeshbalamohan/spark SPARK-12417

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10842.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10842


commit 8615764056bc9039933ca97d85564cf60097fb5a
Author: Rajesh Balamohan 
Date:   2016-01-19T04:16:28Z

SPARK-12417. [SQL] Orc bloom filter options are not propagated during file




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-12898. Consider having dummyCallSite for...

[rajeshbalamohan]

Github user rajeshbalamohan commented on the pull request:

https://github.com/apache/spark/pull/10825#issuecomment-173106715
  
Thanks for review. I have added a comment in the code for the same.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12881] [SQL] subexpress elimination in ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10814#issuecomment-173104792
  
**[Test build #49762 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49762/consoleFull)**
 for PR 10814 at commit 
[`d907949`](https://github.com/apache/spark/commit/d9079498e1c82cd1ed349213b63be78af1ab1bac).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12895] Implement TaskMetrics with accum...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10835#issuecomment-173104844
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49753/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/10837#discussion_r50217890
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ---
@@ -1165,7 +1165,7 @@ class Analyzer(
  * scoping information for attributes and can be removed once analysis is 
complete.
  */
 object EliminateSubQueries extends Rule[LogicalPlan] {
-  def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+  def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
--- End diff --

for future reference, `transformDown` doesn't work because of: 
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala#L267

After apple rule to current node, it goes to its children next. So for a 
nested subquery, like `Sub(Sub(abc))`, it first apply the rule and got 
`Sub(abc)`, then goes to children, i.e. `abc`, so the mid subquery is skipped.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12895] Implement TaskMetrics with accum...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10835#issuecomment-173104840
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/10837#issuecomment-173104610
  
LGTM, pending tests


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12895] Implement TaskMetrics with accum...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10835#issuecomment-173104557
  
**[Test build #49753 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49753/consoleFull)**
 for PR 10835 at commit 
[`308db4c`](https://github.com/apache/spark/commit/308db4cb4712893da43fe8589ee3bbfe31b7958b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10837#issuecomment-173100664
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12895] Implement TaskMetrics with accum...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10835#issuecomment-173100294
  
**[Test build #2418 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2418/consoleFull)**
 for PR 10835 at commit 
[`308db4c`](https://github.com/apache/spark/commit/308db4cb4712893da43fe8589ee3bbfe31b7958b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10837#issuecomment-173100673
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49756/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12881] [SQL] subexpress elimination in ...

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/10814#issuecomment-173100491
  
@nongli done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10837#issuecomment-173099164
  
**[Test build #49756 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49756/consoleFull)**
 for PR 10837 at commit 
[`c4a440f`](https://github.com/apache/spark/commit/c4a440fac8441b37702a3af68ab5a79254e05d7c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10837#issuecomment-173099527
  
**[Test build #49761 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49761/consoleFull)**
 for PR 10837 at commit 
[`93a7728`](https://github.com/apache/spark/commit/93a7728b1cd5623feba4a17c39988c639e6a5b5d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12895] Implement TaskMetrics with accum...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10835#issuecomment-173095034
  
**[Test build #2416 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2416/consoleFull)**
 for PR 10835 at commit 
[`308db4c`](https://github.com/apache/spark/commit/308db4cb4712893da43fe8589ee3bbfe31b7958b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12616] [SQL] Making Logical Operator `U...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10577#issuecomment-173094840
  
**[Test build #49760 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49760/consoleFull)**
 for PR 10577 at commit 
[`c63f237`](https://github.com/apache/spark/commit/c63f237b6de61c700cec3544820674c1dcb94334).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12145][SQL] Command 'Set Role [ADMIN|NO...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10144#issuecomment-173094707
  
Build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12145][SQL] Command 'Set Role [ADMIN|NO...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10144#issuecomment-173094708
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49754/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12145][SQL] Command 'Set Role [ADMIN|NO...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10144#issuecomment-173094611
  
**[Test build #49754 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49754/consoleFull)**
 for PR 10144 at commit 
[`f031c12`](https://github.com/apache/spark/commit/f031c12bb122efb00a3279a95ea07d6416d897e6).
 * This patch **fails Spark unit tests**.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11622][MLLIB] Make LibSVMRelation exten...

[zjffdu]

Github user zjffdu commented on a diff in the pull request:

https://github.com/apache/spark/pull/9595#discussion_r50216785
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/ml/source/libsvm/LibSVMRelationSuite.scala
 ---
@@ -79,4 +79,23 @@ class LibSVMRelationSuite extends SparkFunSuite with 
MLlibTestSparkContext {
 val v = row1.getAs[SparseVector](1)
 assert(v == Vectors.sparse(100, Seq((0, 1.0), (2, 2.0), (4, 3.0
   }
+
+  test("write libsvm data and read it again") {
+val df = sqlContext.read.format("libsvm").load(path)
+val writepath = path + "_2"
+df.write.save(writepath)
+
+val df2 = sqlContext.read.format("libsvm").load(writepath)
+val row1 = df.first()
--- End diff --

This is bug of test code, I am verifying df rather than df2 so that the 
test passed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12153][SPARK-7617][MLlib]add support of...

[ygcao]

Github user ygcao commented on the pull request:

https://github.com/apache/spark/pull/10152#issuecomment-173094302
  
adjusted python doctest format. Did the heavy job of running pyspark test 
locally. Now should be OK.
Finished test(python): pyspark.ml.feature (35s)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12616] [SQL] Making Logical Operator `U...

[gatorsmile]

Github user gatorsmile commented on the pull request:

https://github.com/apache/spark/pull/10577#issuecomment-173093961
  
retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12895] Implement TaskMetrics with accum...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10835#issuecomment-173093642
  
**[Test build #2417 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2417/consoleFull)**
 for PR 10835 at commit 
[`308db4c`](https://github.com/apache/spark/commit/308db4cb4712893da43fe8589ee3bbfe31b7958b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12224][SPARKR] R support for JDBC sourc...

[sun-rui]

Github user sun-rui commented on a diff in the pull request:

https://github.com/apache/spark/pull/10480#discussion_r50216049
  
--- Diff: core/src/main/scala/org/apache/spark/api/r/SerDe.scala ---
@@ -355,6 +355,13 @@ private[spark] object SerDe {
   writeInt(dos, v.length)
   v.foreach(elem => writeObject(dos, elem))
 
+// Handle Properties
--- End diff --

It is possible to construct a map for the properties and pass it to a 
helper function on JVM side. That helper function can convert the map into 
java.util.Properties. The problem is that if there are multiple Scala API 
functions taking java.util.Properties as parameters, we have to create as many 
helper functions as them. So I think we can keep current implementation as is 
for now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12895] Implement TaskMetrics with accum...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10835#issuecomment-173091781
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12895] Implement TaskMetrics with accum...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10835#issuecomment-173091782
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49752/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12895] Implement TaskMetrics with accum...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10835#issuecomment-173091634
  
**[Test build #49752 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49752/consoleFull)**
 for PR 10835 at commit 
[`4ca7328`](https://github.com/apache/spark/commit/4ca7328fd5b0d3b9f6185885f36d95358e9101a3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11622][MLLIB] Make LibSVMRelation exten...

[mengxr]

Github user mengxr commented on a diff in the pull request:

https://github.com/apache/spark/pull/9595#discussion_r50215667
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/ml/source/libsvm/LibSVMRelationSuite.scala
 ---
@@ -79,4 +79,23 @@ class LibSVMRelationSuite extends SparkFunSuite with 
MLlibTestSparkContext {
 val v = row1.getAs[SparseVector](1)
 assert(v == Vectors.sparse(100, Seq((0, 1.0), (2, 2.0), (4, 3.0
   }
+
+  test("write libsvm data and read it again") {
+val df = sqlContext.read.format("libsvm").load(path)
+val writepath = path + "_2"
+df.write.save(writepath)
--- End diff --

Could you help check why it passed tests? It could be a Spark SQL 
feature/bug, e.g., it forces using Parquet format when there are Parquet files 
on the input path.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12224][SPARKR] R support for JDBC sourc...

[sun-rui]

Github user sun-rui commented on a diff in the pull request:

https://github.com/apache/spark/pull/10480#discussion_r50215657
  
--- Diff: R/pkg/R/DataFrame.R ---
@@ -2401,4 +2401,41 @@ setMethod("str",
 cat(paste0("\nDisplaying first ", ncol(localDF), " columns 
only."))
   }
 }
-  })
\ No newline at end of file
+  })
+
+#' Saves the content of the DataFrame to an external database table via 
JDBC
+#'
+#' Additional JDBC database connection properties can be set (...)
+#'
+#' Also, mode is used to specify the behavior of the save operation when
+#' data already exists in the data source. There are four modes: \cr
--- End diff --

is "\cr" intended for Roxygen format?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12629] [SparkR] Fixes for DataFrame sav...

[sun-rui]

Github user sun-rui commented on the pull request:

https://github.com/apache/spark/pull/10580#issuecomment-173091268
  
@NarineK, could you update the write.df() function as well?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12145][SQL] Command 'Set Role [ADMIN|NO...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10144#issuecomment-173091006
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12145][SQL] Command 'Set Role [ADMIN|NO...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10144#issuecomment-173091007
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49755/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12145][SQL] Command 'Set Role [ADMIN|NO...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10144#issuecomment-173090902
  
**[Test build #49755 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49755/consoleFull)**
 for PR 10144 at commit 
[`6436aa8`](https://github.com/apache/spark/commit/6436aa8381c9026d4857d6f3ce9c4baf707a95ae).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/10837#discussion_r50215371
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/EliminateSubQueriesSuite.scala
 ---
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.analysis.EliminateSubQueries
+import org.apache.spark.sql.catalyst.dsl.expressions._
+import org.apache.spark.sql.catalyst.dsl.plans._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.plans.PlanTest
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules._
+
+
+class EliminateSubQueriesSuite extends PlanTest with PredicateHelper {
+
+  object Optimize extends RuleExecutor[LogicalPlan] {
+val batches = Batch("EliminateSubQueries", Once, EliminateSubQueries) 
:: Nil
+  }
+
+  protected def assertEquivalent(e1: Expression, e2: Expression): Unit = {
+val correctAnswer = Project(Alias(e2, "out")() :: Nil, 
OneRowRelation).analyze
+val actual = Optimize.execute(Project(Alias(e1, "out")() :: Nil, 
OneRowRelation).analyze)
+comparePlans(actual, correctAnswer)
+  }
+
+  test("eliminate top level subquery") {
+val input = LocalRelation('a.int, 'b.int)
+val query = Subquery("a", input)
+val optimized = Optimize.execute(query.analyze)
+comparePlans(optimized, input)
+  }
+
+  test("eliminate mid-tree subquery") {
+val input = LocalRelation('a.int, 'b.int)
+val query = Filter(TrueLiteral, Subquery("a", input))
+val optimized = Optimize.execute(query.analyze)
+comparePlans(
+  optimized,
+  Filter(TrueLiteral, LocalRelation('a.int, 'b.int)))
+  }
+
+  test("eliminate multiple subqueries") {
--- End diff --

how about we add one more test that eliminate multiple subqueries from 
multiple children, like join?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7499][SPARKR][WIP] Investigate how to s...

[sun-rui]

Github user sun-rui closed the pull request at:

https://github.com/apache/spark/pull/9835


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7499][SPARKR][WIP] Investigate how to s...

[sun-rui]

Github user sun-rui commented on the pull request:

https://github.com/apache/spark/pull/9835#issuecomment-173089659
  
Close it. Will send new PR later.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...

[napsternxg]

Github user napsternxg commented on a diff in the pull request:

https://github.com/apache/spark/pull/10836#discussion_r50215136
  
--- Diff: R/README.md ---
@@ -1,6 +1,16 @@
 # R on Spark
 
 SparkR is an R package that provides a light-weight frontend to use Spark 
from R.
+### Installing sparkR
+
+Libraries of sparkR need to be created in `$SPARK_HOME/R/lib`. This can be 
done by running the script `$SPARK_HOME/R/install-dev.sh`.
--- End diff --

I think this is needed as mentioned in running sparkR from R studio. The 
script there tries to access the lib location which might not be present by 
default in the sparkR folder if the wrong version of R is selected by default. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Fixes SPARK-12910: R version for installing sp...

[napsternxg]

Github user napsternxg commented on a diff in the pull request:

https://github.com/apache/spark/pull/10836#discussion_r50215117
  
--- Diff: R/README.md ---
@@ -1,6 +1,16 @@
 # R on Spark
 
 SparkR is an R package that provides a light-weight frontend to use Spark 
from R.
+### Installing sparkR
+
+Libraries of sparkR need to be created in `$SPARK_HOME/R/lib`. This can be 
done by running the script `$SPARK_HOME/R/install-dev.sh`.
--- End diff --

I think this is needed as mentioned in running sparkR from R studio. The
script there tries to access the lib location which might not be present by
default in the sparkR folder if the wrong version of R is selected by
default.


*Regards,*
*Shubhanshu Mishra*
Research Assistant,
iSchool at University of Illinois at Urbana-Champaign
--
*Website:* http://shubhanshu.com
*LinkedIn Profile: *http://www.linkedin.com/in/shubhanshumishra

Blog   || Facebook
  ||  Twitter
  || LinkedIn


On Tue, Jan 19, 2016 at 10:03 PM, Felix Cheung 
wrote:

> In R/README.md
> :
>
> > @@ -1,6 +1,16 @@
> >  # R on Spark
> >
> >  SparkR is an R package that provides a light-weight frontend to use 
Spark from R.
> > +### Installing sparkR
> > +
> > +Libraries of sparkR need to be created in `$SPARK_HOME/R/lib`. This 
can be done by running the script `$SPARK_HOME/R/install-dev.sh`.
>
> To clarify, I'm referring the part about "sparkR need to be created in
> $SPARK_HOME/R/lib. This can be done by running the script
> $SPARK_HOME/R/install-dev.sh."
>
> —
> Reply to this email directly or view it on GitHub
> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/10837#discussion_r50214929
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ---
@@ -1165,7 +1165,7 @@ class Analyzer(
  * scoping information for attributes and can be removed once analysis is 
complete.
  */
 object EliminateSubQueries extends Rule[LogicalPlan] {
-  def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+  def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
--- End diff --

No we don't. The previous version of this used just transform and failed 
the tests.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12610][SQL][WIP] Add Anti join operator...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10563#issuecomment-173088692
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49759/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12610][SQL][WIP] Add Anti join operator...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10563#issuecomment-173088685
  
**[Test build #49759 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49759/consoleFull)**
 for PR 10563 at commit 
[`e12ff93`](https://github.com/apache/spark/commit/e12ff93263d9a356b0d51b2ff52914bab7da8e2b).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12689][SQL] Migrate DDL parsing to the ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10723#issuecomment-173088652
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49751/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12610][SQL][WIP] Add Anti join operator...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10563#issuecomment-173088689
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12689][SQL] Migrate DDL parsing to the ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10723#issuecomment-173088651
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10524][ML] Use the soft prediction to o...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8734#issuecomment-173088520
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12610][SQL][WIP] Add Anti join operator...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10563#issuecomment-173088570
  
**[Test build #49759 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49759/consoleFull)**
 for PR 10563 at commit 
[`e12ff93`](https://github.com/apache/spark/commit/e12ff93263d9a356b0d51b2ff52914bab7da8e2b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12689][SQL] Migrate DDL parsing to the ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10723#issuecomment-173088539
  
**[Test build #49751 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49751/consoleFull)**
 for PR 10723 at commit 
[`559083e`](https://github.com/apache/spark/commit/559083eddd0129edeba128927be5810e3a53dc9d).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10524][ML] Use the soft prediction to o...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8734#issuecomment-173088522
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49757/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10524][ML] Use the soft prediction to o...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8734#issuecomment-173088436
  
**[Test build #49757 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49757/consoleFull)**
 for PR 8734 at commit 
[`2c32350`](https://github.com/apache/spark/commit/2c3235054d2d2a98f9f88ae67243bd1ba89eca1e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/10837#discussion_r50214639
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ---
@@ -1165,7 +1165,7 @@ class Analyzer(
  * scoping information for attributes and can be removed once analysis is 
complete.
  */
 object EliminateSubQueries extends Rule[LogicalPlan] {
-  def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+  def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
--- End diff --

we don't recurse into the old node that being replaced, but we do recurse 
into the new node right? Actually I copy and paste your test suite into master 
branch and it passes


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12871][SQL] Support to specify the opti...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/10805


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11518] [Deploy, Windows] Handle spaces ...

[tritab]

Github user tritab commented on the pull request:

https://github.com/apache/spark/pull/10789#issuecomment-173086028
  
I made a number of updates to all of the .cmd files, @tsudukim please 
review to see how it works in your environment. It appears to handle a spark 
install under "Program Files" now. However, I have a couple of outstanding 
concerns that could use some work.

1. If you ctrl-c out of a command, the prompt is left  in the spark 
directory rather than where you started. This isn't optimal.
2. pyspark wasn't working for me, but I have 3.4 installed,which may be 
part of the issue
3. I wasn't able to test R, so that may not be fully functional

Any feedback is welcome. 

@JoshRosen I looked briefly, but wasn't able to find a tool similar to 
shellcheck for Windows. That certainly would be helpful though. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12871][SQL] Support to specify the opti...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/10805#issuecomment-173085873
  
I've merged this in master. Thanks.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/10837#discussion_r50214256
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ---
@@ -1165,7 +1165,7 @@ class Analyzer(
  * scoping information for attributes and can be removed once analysis is 
complete.
  */
 object EliminateSubQueries extends Rule[LogicalPlan] {
-  def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+  def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
--- End diff --

No. See your own comment below.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12204][SPARKR] Implement drop method fo...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10201#issuecomment-173085808
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/10837#discussion_r50214246
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala 
---
@@ -253,6 +256,9 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] 
extends Product {
   /**
* Returns a copy of this node where `rule` has been recursively applied 
to it and all of its
* children (pre-order). When `rule` does not apply to a given node it 
is left unchanged.
+   *
+   * Note that this function does not recurse into the children of a node 
that is being replaced.
--- End diff --

I think so - actually both Michael and I were surprised today by this.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12204][SPARKR] Implement drop method fo...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10201#issuecomment-173085809
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49758/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12204][SPARKR] Implement drop method fo...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10201#issuecomment-173085805
  
**[Test build #49758 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49758/consoleFull)**
 for PR 10201 at commit 
[`89657f8`](https://github.com/apache/spark/commit/89657f8f059ea91b83cace02314b1b6865eea80d).
 * This patch **fails R style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12204][SPARKR] Implement drop method fo...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10201#issuecomment-173085582
  
**[Test build #49758 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49758/consoleFull)**
 for PR 10201 at commit 
[`89657f8`](https://github.com/apache/spark/commit/89657f8f059ea91b83cace02314b1b6865eea80d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11622][MLLIB] Make LibSVMRelation exten...

[zjffdu]

Github user zjffdu commented on a diff in the pull request:

https://github.com/apache/spark/pull/9595#discussion_r50214111
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/ml/source/libsvm/LibSVMRelationSuite.scala
 ---
@@ -79,4 +79,23 @@ class LibSVMRelationSuite extends SparkFunSuite with 
MLlibTestSparkContext {
 val v = row1.getAs[SparseVector](1)
 assert(v == Vectors.sparse(100, Seq((0, 1.0), (2, 2.0), (4, 3.0
   }
+
+  test("write libsvm data and read it again") {
+val df = sqlContext.read.format("libsvm").load(path)
+val writepath = path + "_2"
+df.write.save(writepath)
--- End diff --

Right, this is an  bug of test code. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12567][SQL] Add aes_{encrypt,decrypt} U...

[vectorijk]

Github user vectorijk commented on a diff in the pull request:

https://github.com/apache/spark/pull/10527#discussion_r50214041
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MiscFunctionsSuite.scala
 ---
@@ -130,4 +130,49 @@ class MiscFunctionsSuite extends SparkFunSuite with 
ExpressionEvalHelper {
   }
 }
   }
+
+  test("aesEncrypt") {
+checkEvaluation(Base64(AesEncrypt(Literal("ABC".getBytes),
+  Literal("1234567890123456".getBytes))), "y6Ss+zCYObpCbgfWfyNWTw==")
+checkEvaluation(Base64(AesEncrypt(Literal("".getBytes),
+  Literal("1234567890123456".getBytes))), "BQGHoM3lqYcsurCRq3PlUw==")
+
+// Before testing this, installing Java Cryptography Extension (JCE)
+// Unlimited Strength Jurisdiction Policy Files first
+//checkEvaluation(Base64(AesEncrypt(Literal("ABC".getBytes),
+//  Literal("12345678901234561234567890123456".getBytes))), 
"nYfCuJeRd5eD60yXDw7WEA==")
--- End diff --

@marmbrus When Java Cryptography Extension (JCE) not installed under oracle 
jdk, it will return `null` and will not break other things. Plus, we don't need 
to worry about installing this extension under openjdk, it works fine with 
openjdk.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12699][SPARKR] R driver process should ...

[sun-rui]

Github user sun-rui commented on the pull request:

https://github.com/apache/spark/pull/10652#issuecomment-173084751
  
@felixcheung, yes, something like that


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12204][SPARKR] Implement drop method fo...

[sun-rui]

Github user sun-rui commented on the pull request:

https://github.com/apache/spark/pull/10201#issuecomment-173084587
  
rebased to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12912][SQL] Add a test suite for Elimin...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10837#issuecomment-173084090
  
**[Test build #2415 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2415/consoleFull)**
 for PR 10837 at commit 
[`c2dba6a`](https://github.com/apache/spark/commit/c2dba6a22451652c97a323270b66781cabe15256).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10524][ML] Use the soft prediction to o...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8734#issuecomment-173080387
  
**[Test build #49757 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49757/consoleFull)**
 for PR 8734 at commit 
[`2c32350`](https://github.com/apache/spark/commit/2c3235054d2d2a98f9f88ae67243bd1ba89eca1e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12871][SQL] Support to specify the opti...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10805#issuecomment-173080338
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org