[GitHub] spark pull request: [SPARK-2096][SQL] Correctly parse dot notation...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2230#issuecomment-55077709
  
Jenkins, test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3395] [SQL] DSL sometimes incorrectly r...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2266#issuecomment-55077558
  
Thanks! Merged to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3395] [SQL] DSL sometimes incorrectly r...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/2266


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3453] [WIP] Refactor Netty module to us...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/2330#issuecomment-55077449
  
@colorant you probably would be interested in this ...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3030] [PySpark] Reuse Python worker

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/2259#issuecomment-55077384
  
@JoshRosen The problem that will cause hanging has been fixed. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3433][BUILD] Fix for Mima false-positiv...

[ScrapCodes]

Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/2285#issuecomment-55077314
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3030] [PySpark] Reuse Python worker

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/2259#issuecomment-55077329
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] Add test case with workaround for readin...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2340#issuecomment-55076938
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3447][SQL] Remove explicit conversion w...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2323#issuecomment-55076971
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2890][SQL] Allow reading of data when c...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2209#issuecomment-55076950
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3469] Call all TaskCompletionListeners ...

[rxin]

GitHub user rxin opened a pull request:

https://github.com/apache/spark/pull/2343

[SPARK-3469] Call all TaskCompletionListeners even if some fail

This is necessary because we rely on this callback interface to clean 
resources up. The old behavior would lead to resource leaks.

Note that this also changes the fault semantics of TaskCompletionListener.
Previously failures in TaskCompletionListeners would result in the task
being reported as failed. With this change, tasks won't be reported as 
failed
simply because the execution of TaskCompletionListener fails.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rxin/spark taskcontext-callback

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/2343.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2343


commit 1cb444d4afc5355625d26fee7bd40ecb5532b8ca
Author: Reynold Xin 
Date:   2014-09-10T06:32:37Z

[SPARK-3469] Call all TaskCompletionListeners even if some fail.

Note that this also changes the fault semantics of TaskCompletionListener.
Previously failures in TaskCompletionListeners would result in the task
being reported as failed. With this change, tasks won't be reported as 
failed
simply because the execution of TaskCompletionListener fails.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3325] Add a parameter to the method pri...

[watermen]

Github user watermen commented on a diff in the pull request:

https://github.com/apache/spark/pull/2216#discussion_r17344850
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaDStreamLike.scala
 ---
@@ -50,8 +50,8 @@ trait JavaDStreamLike[T, This <: JavaDStreamLike[T, This, 
R], R <: JavaRDDLike[T
* Print the first ten elements of each RDD generated in this DStream. 
This is an output
* operator, so this DStream will be registered as an output stream and 
there materialized.
*/
-  def print(): Unit = {
-dstream.print()
+  def print(num: Int = 10): Unit = {
--- End diff --

What do you mean of "break binary compatibility"?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3437][BUILD] Support crossbuilding in m...

[ScrapCodes]

Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/2318#issuecomment-55075420
  
Hey All,
Please correct me if I am wrong, my conclusion is that this also will not 
help because pom published is not appropriate(Since it does not have dependency 
info, which is I guess needed by downstream projects). So IMO writing a plugin 
is the only option now.  I have started writing one here 
https://github.com/ScrapCodes/scala-install-plugin. It is cool in the sense, it 
publishes effective pom that is: if artifact was build using a profile its 
effective pom of dependencies(will have correct properties values set in etc..) 
will be different than one published with a different profile. I am not expert 
here and totally confused, my only source of information is seeing what others 
have done.   


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3452] Maven build should skip publishin...

[ScrapCodes]

Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/2329#issuecomment-55074587
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2917] [SQL] Avoid table creation in log...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1846#issuecomment-55073843
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20082/consoleFull)
 for   PR 1846 at commit 
[`6521ce8`](https://github.com/apache/spark/commit/6521ce8c80d1fd90e2715952380a40dabcad9005).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2342#issuecomment-55073832
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20080/consoleFull)
 for   PR 2342 at commit 
[`e660878`](https://github.com/apache/spark/commit/e6608787659d00370d066e83f714c4bb4e6c0d32).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3465] fix task metrics aggregation in l...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2338#issuecomment-55073847
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20081/consoleFull)
 for   PR 2338 at commit 
[`754b5b8`](https://github.com/apache/spark/commit/754b5b8cefd953e10c78519ba4c6d925c505bc6c).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3007][SQL]Add "Dynamic Partition" suppo...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2226#issuecomment-55073565
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20076/consoleFull)
 for   PR 2226 at commit 
[`406d80a`](https://github.com/apache/spark/commit/406d80ad21fbd86ec38029c067cd5ac3c2ce13c0).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3160] [mllib] DecisionTree: eliminate p...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2341#issuecomment-55073529
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20078/consoleFull)
 for   PR 2341 at commit 
[`306120f`](https://github.com/apache/spark/commit/306120fc93021f3d2d86333c77296fe3d36b76e1).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP][SPARK-3468] WebUI Timeline-View feature

[sarutak]

GitHub user sarutak opened a pull request:

https://github.com/apache/spark/pull/2342

[WIP][SPARK-3468] WebUI Timeline-View feature

I sometimes trouble-shoot and analyse the cause of long time spending job.

At the time, I find the stages which spends long time or fails, then I find 
the tasks which spends long time or fails, next I analyse the proportion of 
each phase in a task.

Another case, I find executors which spends long time for running a task 
and analyse the details of a task.

In such situation, I think it's helpful to visualize timeline view of 
stages / tasks / executors and visualize details of proportion of activity for 
each task.

Now I'm developing prototypes like captures I attached.
I'll integrate these viewer into WebUI.

[Capture of prototype for Stages timeline view]
Each brown bar represents submission time to finish time of a stage.


![stages](https://cloud.githubusercontent.com/assets/4736016/4212893/e1cb20c0-38a8-11e4-91bc-a20042b109fb.png)

[Capture of prototype for Task timeline view]
Each brown bar represents launch time to finish time of a task.


![tasks](https://cloud.githubusercontent.com/assets/4736016/4212914/4f201432-38a9-11e4-91e9-9c2030991468.png)

[Capture of prorotype for Task details view]
This shows the proportion of activities in a task.


![taskdetails](https://cloud.githubusercontent.com/assets/4736016/4212921/74943fe0-38a9-11e4-86ac-8f9f9f6e65c7.png)

[Capture of prorotype for Executor timeline view]
Each brown bar represents start time to finish time of a task which a 
executor executes.


![executors](https://cloud.githubusercontent.com/assets/4736016/4212925/a3bb78ec-38a9-11e4-8909-9d662d8d6093.png)




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sarutak/spark timeline-viewer-feature

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/2342.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2342


commit 4569a8e3ff703c17d125a6a80fff7f6baeb9e1f1
Author: Kousuke Saruta 
Date:   2014-09-10T05:09:35Z

Added d3.min.js

Modified .rat-exclude for d3.min.js

Modified UIUtils.scala for d3.min.js




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2917] [SQL] Avoid table creation in log...

[chenghao-intel]

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/1846#issuecomment-55071879
  
test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-1713. Use a thread pool for launching ex...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/663#issuecomment-55071283
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20077/consoleFull)
 for   PR 663 at commit 
[`036550d`](https://github.com/apache/spark/commit/036550d5354cdf1d15c64fee8d12a581a355dc68).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3465] fix task metrics aggregation in l...

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/2338#issuecomment-55070918
  
@sryza I had changed it to do the copy in Executor, then it's hard to write 
a test now. Any idea?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3160] [mllib] DecisionTree: eliminate p...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2341#issuecomment-55070870
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20078/consoleFull)
 for   PR 2341 at commit 
[`306120f`](https://github.com/apache/spark/commit/306120fc93021f3d2d86333c77296fe3d36b76e1).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] Add test case with workaround for readin...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2340#issuecomment-55070874
  
**[Tests timed 
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20068/consoleFull)**
 after a configured wait of `120m`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3447][SQL] Remove explicit conversion w...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2323#issuecomment-55070867
  
**[Tests timed 
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20070/consoleFull)**
 after a configured wait of `120m`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3395] [SQL] DSL sometimes incorrectly r...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2266#issuecomment-55070828
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20073/consoleFull)
 for   PR 2266 at commit 
[`7f2b6f0`](https://github.com/apache/spark/commit/7f2b6f000cdcbe00ed138e96c0cca5bd0623a705).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3294][SQL] WIP: eliminates boxing costs...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2327#issuecomment-55069727
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20072/consoleFull)
 for   PR 2327 at commit 
[`97bbc4e`](https://github.com/apache/spark/commit/97bbc4ed9b9e3f1b8248db78bd57b6b33a09f2c0).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2781][SQL] Check resolution of LogicalP...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1706#issuecomment-55069715
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20075/consoleFull)
 for   PR 1706 at commit 
[`80a27dc`](https://github.com/apache/spark/commit/80a27dcf48f9c51c64b17ac147abde950c4d2096).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class Last(child: Expression) extends PartialAggregate with 
trees.UnaryNode[Expression] `
  * `case class LastFunction(expr: Expression, base: AggregateExpression) 
extends AggregateFunction `
  * `case class Abs(child: Expression) extends UnaryExpression  `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2207][SPARK-3272][MLLib]Add minimum inf...

[jkbradley]

Github user jkbradley commented on the pull request:

https://github.com/apache/spark/pull/2332#issuecomment-55069323
  
@chouqin   Thanks for the updates!  This looks basically ready, except for 
the edge cases in the test suite.  I tested it and it ran fine.  I think those 
complaints about public classes are unrelated.  Once the test suite is updated, 
I'd say it is ready.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2207][SPARK-3272][MLLib]Add minimum inf...

[jkbradley]

Github user jkbradley commented on a diff in the pull request:

https://github.com/apache/spark/pull/2332#discussion_r17342007
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/mllib/tree/DecisionTreeSuite.scala ---
@@ -674,6 +676,45 @@ class DecisionTreeSuite extends FunSuite with 
LocalSparkContext {
 validateClassifier(model, arr, 0.6)
   }
 
+  test("split must satisfy min instances per node requirements") {
--- End diff --

Yes, that was what I had in mind.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3160] [mllib] DecisionTree: eliminate p...

[jkbradley]

GitHub user jkbradley opened a pull request:

https://github.com/apache/spark/pull/2341

[SPARK-3160] [mllib]  DecisionTree: eliminate pre-allocated nodes, 
parentImpurities arrays

This PR includes some code simplifications and re-organization which will 
be helpful for implementing random forests.  The main changes are that the 
nodes and parentImpurities arrays are no longer pre-allocated in the main 
train() method.

Relation to RFs:
* Since RFs will be deeper and will therefore be more likely sparse (not 
full trees), it could be a cost savings to avoid pre-allocating a full tree.
* The associated re-organization also reduces bookkeeping, which will make 
RFs easier to implement.
* The return code doneTraining may be generalized to include cases such as 
nodes ready for local training.

Details:

No longer pre-allocate parentImpurities array in main train() method.
* parentImpurities values are now stored in individual nodes (in 
Node.stats.impurity).
* These were not really needed.  They were used in calculateGainForSplit(), 
but they can be calculated anyways using parentNodeAgg.

No longer using Node.build since tree structure is constructed on-the-fly.
* Did not eliminate since it is public (Developer) API.  Marked as 
deprecated.

Eliminated pre-allocated nodes array in main train() method.
* Nodes are constructed and added to the tree structure as needed during 
training.
* Moved tree construction from main train() method into 
findBestSplitsPerGroup() since there is no need to keep the (split, gain) array 
for an entire level of nodes.  Only one element of that array is needed at a 
time, so we do not the array.

findBestSplits() now returns 2 items:
* rootNode (newly created root node on first iteration, same root node on 
later iterations)
* doneTraining (indicating if all nodes at that level were leafs)

Updated DecisionTreeSuite.  Notes:
* Improved test "Second level node building with vs. without groups"
** generateOrderedLabeledPoints() modified so that it really does require 2 
levels of internal nodes.
* Related update: Added Node.deepCopy (private[tree]), used for test suite

CC: @mengxr

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkbradley/spark dt-spark-3160

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/2341.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2341


commit 2ab763b2ca1bbc897ab898b28965dce5a8a3
Author: Joseph K. Bradley 
Date:   2014-09-09T17:42:46Z

Simplifications to DecisionTree code:

No longer pre-allocate parentImpurities array in main train() method.
* parentImpurities values are now stored in individual nodes (in 
Node.stats.impurity).

No longer using Node.build since tree structure is constructed on-the-fly.
* Did not eliminate since it is public (Developer) API.

Also: Updated DecisionTreeSuite test "Second level node building with vs. 
without groups"
* generateOrderedLabeledPoints() modified so that it really does require 2 
levels of internal nodes.

commit 1a8f0add470e4ed53100ce6cf344e24448a0ba42
Author: Joseph K. Bradley 
Date:   2014-09-10T02:34:55Z

Eliminated pre-allocated nodes array in main train() method.
* Nodes are constructed and added to the tree structure as needed during 
training.

Moved tree construction from main train() method into 
findBestSplitsPerGroup() since there is no need to keep the (split, gain) array 
for an entire level of nodes.  Only one element of that array is needed at a 
time, so we do not the array.

findBestSplits() now returns 2 items:
* rootNode (newly created root node on first iteration, same root node on 
later iterations)
* doneTraining (indicating if all nodes at that level were leafs)

Also:
* Added Node.deepCopy (private[tree]), used for test suite
* Updated test suite (same functionality)

commit d4dbb99a50418e0168d85db457458d8d96edc242
Author: Joseph K. Bradley 
Date:   2014-09-10T02:35:06Z

Merge remote-tracking branch 'upstream/master' into dt-spark-3160

commit d4d786407a9bb5fce14dd7999097b21d6fa1cf5e
Author: Joseph K. Bradley 
Date:   2014-09-10T02:45:30Z

Marked Node.build as deprecated

commit eaa1dcf6a46501779ae58c746e672583d10ff6c8
Author: Joseph K. Bradley 
Date:   2014-09-10T02:58:27Z

Added topNode doc in DecisionTree and scalastyle fix

commit 306120fc93021f3d2d86333c77296fe3d36b76e1
Author: Joseph K. Bradley 
Date:   2014-09-10T03:09:02Z

Fixed typo in DecisionTreeModel.scala doc




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this featur

[GitHub] spark pull request: [SPARK-3160] [mllib] DecisionTree: eliminate p...

[jkbradley]

Github user jkbradley commented on the pull request:

https://github.com/apache/spark/pull/2341#issuecomment-55068847
  
I will wait until [https://github.com/apache/spark/pull/2332] is merged, 
and then will update this with the merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3430] [PySpark] [Doc] generate PySpark ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2292#issuecomment-55068634
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20074/consoleFull)
 for   PR 2292 at commit 
[`9468ab0`](https://github.com/apache/spark/commit/9468ab0cc210f444fbc18ebd34dc99ba19636499).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2096][SQL] Correctly parse dot notation...

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/2230#issuecomment-55068488
  
rebase done, test again please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor] rat exclude dependency-reduced-pom.xml

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2326#issuecomment-55068309
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20071/consoleFull)
 for   PR 2326 at commit 
[`860904e`](https://github.com/apache/spark/commit/860904e96c7a4e06adc80e36163891f9b6f9175d).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class Last(child: Expression) extends PartialAggregate with 
trees.UnaryNode[Expression] `
  * `case class LastFunction(expr: Expression, base: AggregateExpression) 
extends AggregateFunction `
  * `case class Abs(child: Expression) extends UnaryExpression  `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3030] [PySpark] Reuse Python worker

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/2259#issuecomment-55068303
  
yeah, I will investigate it locally.

On Tue, Sep 9, 2014 at 8:53 PM, Josh Rosen  wrote:

> Hmm, I wonder why we're seeing these timeouts. It looks like both tests
> failed in recommendation.py, so it might be worth running those tests
> locally to see whether they're running way slower after this patch.
>
> --
> Reply to this email directly or view it on GitHub
> .
>



-- 
 - Davies


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2207][SPARK-3272][MLLib]Add minimum inf...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2332#issuecomment-55068261
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20069/consoleFull)
 for   PR 2332 at commit 
[`0278a11`](https://github.com/apache/spark/commit/0278a1198017aae578be3109a8311abc1f9a8e14).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class Last(child: Expression) extends PartialAggregate with 
trees.UnaryNode[Expression] `
  * `case class LastFunction(expr: Expression, base: AggregateExpression) 
extends AggregateFunction `
  * `case class Abs(child: Expression) extends UnaryExpression  `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3030] [PySpark] Reuse Python worker

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/2259#issuecomment-55068149
  
Hmm, I wonder why we're seeing these timeouts.  It looks like both tests 
failed in `recommendation.py`, so it might be worth running those tests locally 
to see whether they're running way slower after this patch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3007][SQL]Add "Dynamic Partition" suppo...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2226#issuecomment-55068049
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20076/consoleFull)
 for   PR 2226 at commit 
[`406d80a`](https://github.com/apache/spark/commit/406d80ad21fbd86ec38029c067cd5ac3c2ce13c0).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-1713. Use a thread pool for launching ex...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/663#issuecomment-55068043
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20077/consoleFull)
 for   PR 663 at commit 
[`036550d`](https://github.com/apache/spark/commit/036550d5354cdf1d15c64fee8d12a581a355dc68).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3030] [PySpark] Reuse Python worker

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2259#issuecomment-55068034
  
**[Tests timed 
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20062/consoleFull)**
 after a configured wait of `120m`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3362][SQL] bug in casewhen resolve

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2245#issuecomment-55067493
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20064/consoleFull)
 for   PR 2245 at commit 
[`3332f6e`](https://github.com/apache/spark/commit/3332f6e3570cd17dd3aa55c3254ce56692e4a502).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class Last(child: Expression) extends PartialAggregate with 
trees.UnaryNode[Expression] `
  * `case class LastFunction(expr: Expression, base: AggregateExpression) 
extends AggregateFunction `
  * `case class Abs(child: Expression) extends UnaryExpression  `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3421][SQL] Allows arbitrary character i...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2291#issuecomment-55067504
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20061/consoleFull)
 for   PR 2291 at commit 
[`f3d8c98`](https://github.com/apache/spark/commit/f3d8c98c3360220a5308eedf915c5772ff91a9fb).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class Last(child: Expression) extends PartialAggregate with 
trees.UnaryNode[Expression] `
  * `case class LastFunction(expr: Expression, base: AggregateExpression) 
extends AggregateFunction `
  * `case class Abs(child: Expression) extends UnaryExpression  `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3007][SQL]Add "Dynamic Partition" suppo...

[baishuo]

Github user baishuo commented on the pull request:

https://github.com/apache/spark/pull/2226#issuecomment-55067304
  
had update the file according liancheng's comment. and test it locally


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3363][SQL] Type Coercion should support...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2246#issuecomment-55066934
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20063/consoleFull)
 for   PR 2246 at commit 
[`c6241de`](https://github.com/apache/spark/commit/c6241de74b75a750be89ba4b1635e388781d8bcf).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class Last(child: Expression) extends PartialAggregate with 
trees.UnaryNode[Expression] `
  * `case class LastFunction(expr: Expression, base: AggregateExpression) 
extends AggregateFunction `
  * `case class Abs(child: Expression) extends UnaryExpression  `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3465] fix task metrics aggregation in l...

[sryza]

Github user sryza commented on the pull request:

https://github.com/apache/spark/pull/2338#issuecomment-55066419
  
Hi @davies , sorry for causing this bug and thanks for picking it up.  To 
avoid making the deep copy unnecessarily when running in non-local mode, we 
could instead make it on the executor side, and only do so if isLocal = true.  
Any issues you can see with that?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-1713. Use a thread pool for launching ex...

[sryza]

Github user sryza commented on the pull request:

https://github.com/apache/spark/pull/663#issuecomment-55066027
  
Upmerged


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2781][SQL] Check resolution of LogicalP...

[staple]

Github user staple commented on the pull request:

https://github.com/apache/spark/pull/1706#issuecomment-55066004
  
Ok, sure I'll move ExtractPythonUdfs as you suggest. Thanks for taking a 
look!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] Decrease partitions when testing

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2164#issuecomment-55065939
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20066/consoleFull)
 for   PR 2164 at commit 
[`ee687cd`](https://github.com/apache/spark/commit/ee687cd1eb0c8d89acbfc176cb822f43a9bd4c2a).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class Last(child: Expression) extends PartialAggregate with 
trees.UnaryNode[Expression] `
  * `case class LastFunction(expr: Expression, base: AggregateExpression) 
extends AggregateFunction `
  * `case class Abs(child: Expression) extends UnaryExpression  `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2781][SQL] Check resolution of LogicalP...

[staple]

Github user staple commented on a diff in the pull request:

https://github.com/apache/spark/pull/1706#discussion_r17340821
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ---
@@ -40,7 +40,12 @@ class Analyzer(catalog: Catalog, registry: 
FunctionRegistry, caseSensitive: Bool
   // TODO: pass this in as a parameter.
   val fixedPoint = FixedPoint(100)
 
-  val batches: Seq[Batch] = Seq(
+  /**
+   * Override to provide additional rules for the "Resolution" batch.
+   */
+  val extendedRules: List[Rule[LogicalPlan]] = Nil
--- End diff --

I'd made it a List in order to use the triple colon concat operator in 
Analyzer for consistency with the existing code that uses double colon, but 
sure I can change to Seq instead since that's preferred.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3398] [EC2] Have spark-ec2 intelligentl...

[nchammas]

Github user nchammas commented on the pull request:

https://github.com/apache/spark/pull/2339#issuecomment-55065697
  
@JoshRosen and @davies: This PR is ready for review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3411] Improve load-balancing of concurr...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1106#issuecomment-55065665
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20067/consoleFull)
 for   PR 1106 at commit 
[`d1a928b`](https://github.com/apache/spark/commit/d1a928b3afaa32ded9dd2e087da4b1229028fdba).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class Last(child: Expression) extends PartialAggregate with 
trees.UnaryNode[Expression] `
  * `case class LastFunction(expr: Expression, base: AggregateExpression) 
extends AggregateFunction `
  * `case class Abs(child: Expression) extends UnaryExpression  `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-2461. Add a toString method to Generaliz...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1388#issuecomment-55064976
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20065/consoleFull)
 for   PR 1388 at commit 
[`e6f29cd`](https://github.com/apache/spark/commit/e6f29cd0b70cdd2bc81ddf7ac01db456b16cc55f).
 * This patch **fails** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class Last(child: Expression) extends PartialAggregate with 
trees.UnaryNode[Expression] `
  * `case class LastFunction(expr: Expression, base: AggregateExpression) 
extends AggregateFunction `
  * `case class Abs(child: Expression) extends UnaryExpression  `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] Add test case with workaround for readin...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2340#issuecomment-55064959
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20068/consoleFull)
 for   PR 2340 at commit 
[`6b969d6`](https://github.com/apache/spark/commit/6b969d67b043d0dc5b7a4c530b5ca9c4783f3fbf).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3395] [SQL] DSL sometimes incorrectly r...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2266#issuecomment-55064953
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20073/consoleFull)
 for   PR 2266 at commit 
[`7f2b6f0`](https://github.com/apache/spark/commit/7f2b6f000cdcbe00ed138e96c0cca5bd0623a705).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3294][SQL] WIP: eliminates boxing costs...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2327#issuecomment-55064948
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20072/consoleFull)
 for   PR 2327 at commit 
[`97bbc4e`](https://github.com/apache/spark/commit/97bbc4ed9b9e3f1b8248db78bd57b6b33a09f2c0).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2781][SQL] Check resolution of LogicalP...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1706#issuecomment-55064956
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20075/consoleFull)
 for   PR 1706 at commit 
[`80a27dc`](https://github.com/apache/spark/commit/80a27dcf48f9c51c64b17ac147abde950c4d2096).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2207][SPARK-3272][MLLib]Add minimum inf...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2332#issuecomment-55064961
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20069/consoleFull)
 for   PR 2332 at commit 
[`0278a11`](https://github.com/apache/spark/commit/0278a1198017aae578be3109a8311abc1f9a8e14).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3447][SQL] Remove explicit conversion w...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2323#issuecomment-55064952
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20070/consoleFull)
 for   PR 2323 at commit 
[`646976b`](https://github.com/apache/spark/commit/646976b1912b9f7c299b739db4cf49131c8b28eb).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor] rat exclude dependency-reduced-pom.xml

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2326#issuecomment-55064937
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20071/consoleFull)
 for   PR 2326 at commit 
[`860904e`](https://github.com/apache/spark/commit/860904e96c7a4e06adc80e36163891f9b6f9175d).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3030] [PySpark] Reuse Python worker

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2259#issuecomment-55064927
  
**[Tests timed 
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20060/consoleFull)**
 after a configured wait of `120m`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3430] [PySpark] [Doc] generate PySpark ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2292#issuecomment-55064938
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20074/consoleFull)
 for   PR 2292 at commit 
[`9468ab0`](https://github.com/apache/spark/commit/9468ab0cc210f444fbc18ebd34dc99ba19636499).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3040] pick up a more proper local ip ad...

[advancedxy]

Github user advancedxy commented on the pull request:

https://github.com/apache/spark/pull/1946#issuecomment-55064727
  
I'd love to see this get merged into 1.2. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2917] [SQL] Avoid table creation in log...

[chenghao-intel]

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/1846#issuecomment-55064532
  
Thank you @marmbrus I will add an unit test for this soon. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Minor] rat exclude dependency-reduced-pom.xml

[witgo]

Github user witgo commented on the pull request:

https://github.com/apache/spark/pull/2326#issuecomment-55064478
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] Add test case with workaround for readin...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2340#issuecomment-55064281
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3430] [PySpark] [Doc] generate PySpark ...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2292#issuecomment-55064229
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2781][SQL] Check resolution of LogicalP...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/1706#issuecomment-55064204
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3395] [SQL] DSL sometimes incorrectly r...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2266#issuecomment-55064245
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2781][SQL] Check resolution of LogicalP...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/1706#issuecomment-55064192
  
Oh, hmmm, its actually probably a bug that `ExtractPythonUdfs` is not run 
to fixed point.  I think it is safe for you to add it using `extendedRules` 
instead of how its done now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2781][SQL] Check resolution of LogicalP...

[marmbrus]

Github user marmbrus commented on a diff in the pull request:

https://github.com/apache/spark/pull/1706#discussion_r17340248
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 ---
@@ -40,7 +40,12 @@ class Analyzer(catalog: Catalog, registry: 
FunctionRegistry, caseSensitive: Bool
   // TODO: pass this in as a parameter.
   val fixedPoint = FixedPoint(100)
 
-  val batches: Seq[Batch] = Seq(
+  /**
+   * Override to provide additional rules for the "Resolution" batch.
+   */
+  val extendedRules: List[Rule[LogicalPlan]] = Nil
--- End diff --

In general I'd use the more general `Seq` instead of `List` for any 
declared interfaces.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2207][SPARK-3272][MLLib]Add minimum inf...

[chouqin]

Github user chouqin commented on the pull request:

https://github.com/apache/spark/pull/2332#issuecomment-55063959
  
@jkbradley I have removed `noSplit` object and add `private[tree]` to 
`Predict`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3421][SQL] Allows arbitrary character i...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2291#discussion_r17340060
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataTypeSuite.scala 
---
@@ -55,4 +55,38 @@ class DataTypeSuite extends FunSuite {
   struct(Set("b", "d", "e", "f"))
 }
   }
+
+  test("StructField.toString") {
+def structFieldWithName(name: String) = StructField(name, StringType, 
nullable = true)
+
+assertResult("""StructField("a",StringType,true)""") {
+  structFieldWithName("a").toString
+}
+
+assertResult("""StructField("(a)",StringType,true)""") {
+  structFieldWithName("(a)").toString
+}
+
+assertResult("""StructField("a\\b\"",StringType,true)""") {
+  structFieldWithName("""a\b).toString
+}
+  }
+
+  test("parsing StructField string") {
+val expected = StructType(
+  StructField("a", StringType, true) ::
+  StructField("\"b\"", StringType, true) ::
+  StructField("\"c\\", StringType, true) ::
+  Nil)
+
+val structTypeString = Seq(
+  """StructType(List(""",
+  """StructField("a",StringType,true),""",
+  """StructField("\"b\"",StringType,true),""",
+  """StructField("\"c\\",StringType,true)""",
+  """))"""
+).mkString
+
+assert(catalyst.types.DataType(structTypeString) === expected)
--- End diff --

OK, some ScalaCheck style random generated input may be helpful here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2917] [SQL] Avoid table creation in log...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/1846#issuecomment-55063318
  
@chenghao-intel, I'd like to this merged before I begin on my next big 
refactoring.  One small thing, do you think you could add a test to make sure 
we don't regress and start creating tables for EXPLAIN queries?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3455] [SQL] **HOT FIX** Fix the unit te...

[chenghao-intel]

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/2334#issuecomment-55062668
  
That's cool.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2096][SQL] Correctly parse dot notation...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2230#issuecomment-55062642
  
Yeah I think the test failure was unrelated, though unfortunately this is 
out of date again.  Mind updating one more time?  Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3412] [SQL] Add 3 missing types for Row...

[chenghao-intel]

Github user chenghao-intel closed the pull request at:

https://github.com/apache/spark/pull/2284


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3412] [SQL] Add 3 missing types for Row...

[chenghao-intel]

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/2284#issuecomment-55062524
  
ok, I am closing it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3393] [SQL] add configuration template ...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2263#issuecomment-55062500
  
Oh, okay thanks for explaining this!  My inclination would be to update our 
code to read from `log4j.properties` instead, unless there is some compelling 
reason not to.  Otherwise, can we maybe add a comment to the template file that 
explains what it is for and why its needed in addition to the standard config 
file?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3395] [SQL] DSL sometimes incorrectly r...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2266#issuecomment-55062357
  
Jenkins, test this please



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3430] [PySpark] [Doc] generate PySpark ...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2292#issuecomment-55062317
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] Add test case with workaround for readin...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2340#issuecomment-55062259
  
Jenkins, test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3447][SQL] Remove explicit conversion w...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2323#issuecomment-55061963
  
I've updated the usage in JSON RDD.  Java Row wrapping should never happen 
before Kryo serialization AFAICT.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3447][SQL] Remove explicit conversion w...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2323#issuecomment-55061969
  
Jenkins, test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2207][SPARK-3272][MLLib]Add minimum inf...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2332#issuecomment-55061865
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20058/consoleFull)
 for   PR 2332 at commit 
[`d593ec7`](https://github.com/apache/spark/commit/d593ec70d70b633b72e260c38e89d87ab14fcd69).
 * This patch **passes** unit tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class Predict(`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3458] enable python "with" statements f...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/2335


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3458] enable python "with" statements f...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/2335#issuecomment-55061687
  
Merging in master. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2643] Stages web ui has ERROR when pool...

[jkbradley]

Github user jkbradley commented on the pull request:

https://github.com/apache/spark/pull/1854#issuecomment-55061571
  
@YanTangZhai  Hi, I was asked to check out this PR.  It looks like it may 
no longer be necessary since stages are now paired with attempt IDs.  Could you 
please try running the test which failed with the old Spark version, and see if 
it still fails?  Hopefully the issue is fixed, so that the PR can be closed.  
Thanks very much!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-2461. Add a toString method to Generaliz...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1388#issuecomment-55061444
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20065/consoleFull)
 for   PR 1388 at commit 
[`e6f29cd`](https://github.com/apache/spark/commit/e6f29cd0b70cdd2bc81ddf7ac01db456b16cc55f).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3030] [PySpark] Reuse Python worker

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2259#issuecomment-55061459
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20062/consoleFull)
 for   PR 2259 at commit 
[`cf1c55e`](https://github.com/apache/spark/commit/cf1c55e4fe891742bd2102606b901cc834a87557).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3362][SQL] bug in casewhen resolve

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2245#issuecomment-55061471
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20064/consoleFull)
 for   PR 2245 at commit 
[`3332f6e`](https://github.com/apache/spark/commit/3332f6e3570cd17dd3aa55c3254ce56692e4a502).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3411] Improve load-balancing of concurr...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1106#issuecomment-55061487
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20067/consoleFull)
 for   PR 1106 at commit 
[`d1a928b`](https://github.com/apache/spark/commit/d1a928b3afaa32ded9dd2e087da4b1229028fdba).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3363][SQL] Type Coercion should support...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2246#issuecomment-55061450
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20063/consoleFull)
 for   PR 2246 at commit 
[`c6241de`](https://github.com/apache/spark/commit/c6241de74b75a750be89ba4b1635e388781d8bcf).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] Decrease partitions when testing

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2164#issuecomment-55061458
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20066/consoleFull)
 for   PR 2164 at commit 
[`ee687cd`](https://github.com/apache/spark/commit/ee687cd1eb0c8d89acbfc176cb822f43a9bd4c2a).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3421][SQL] Allows arbitrary character i...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2291#issuecomment-55061434
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20061/consoleFull)
 for   PR 2291 at commit 
[`f3d8c98`](https://github.com/apache/spark/commit/f3d8c98c3360220a5308eedf915c5772ff91a9fb).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3294][SQL] WIP: eliminates boxing costs...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2327#issuecomment-55061419
  
@aarondav to expand on that, as soon as there is any nesting all of our 
clever tricks for eliminating allocations go out the window.  We can probably 
improve this in future releases.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3412] [SQL] Add 3 missing types for Row...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2284#issuecomment-55061149
  
The type for each SQL type are pretty well documented in the programming 
guide (updated for 1.1 to be published soon).  It seems unscalable to add new 
methods to all the various row implementations for each new datatype, 
especially since all they are doing is casting.  Given this I propose we close 
this issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] Decrease partitions when testing

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2164#issuecomment-55060922
  
Jenkins, test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3421][SQL] Allows arbitrary character i...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/2291#issuecomment-55060899
  
Jenkins, test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org