[GitHub] spark pull request: [SPARK-2652] [PySpark] donot use KyroSerialize...

[mengxr]

Github user mengxr commented on the pull request:

https://github.com/apache/spark/pull/2916#issuecomment-60351266
  
LGTM. Merged into both master and branch-1.1. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3886] [PySpark] simplify serializer, us...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2920#issuecomment-60351159
  
  [Test build #22117 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22117/consoleFull)
 for   PR 2920 at commit 
[`8d77ef2`](https://github.com/apache/spark/commit/8d77ef20276af479663725ab6dd70c93e9973bb4).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3886] [PySpark] simplify serializer, us...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2920#issuecomment-60351167
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22117/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3218, SPARK-3219, SPARK-3261, SPARK-342...

[mengxr]

Github user mengxr commented on the pull request:

https://github.com/apache/spark/pull/2634#issuecomment-60351026
  
@derrickburns The features are useful, so please don't delete the PR. Since 
this is a major refactor of `KMeans`, I need to allocate a block of time to 
review the code and run performance tests. But I'm running out of bandwidth 
now. Do you mind me doing the review after the v1.2 feature freeze?

Btw, it would be really helpful if you can post some performance testing 
results against the current implementation in MLlib, for large-scale sparse and 
dense data.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4030] Make destroy public for broadcast...

[pwendell]

Github user pwendell commented on the pull request:

https://github.com/apache/spark/pull/2922#issuecomment-60350983
  
oh one thing - can we add a java version of this? should be pretty simple, 
right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4030] Make destroy public for broadcast...

[pwendell]

Github user pwendell commented on the pull request:

https://github.com/apache/spark/pull/2922#issuecomment-60350936
  
small question - this looks good overall


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

[ScrapCodes]

Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/2615#issuecomment-60350893
  
Not sure why compiler is consistently crashing here. The build 
compilation(and tests) passes locally for both scala 2.10 and scala 2.11. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4030] Make destroy public for broadcast...

[pwendell]

Github user pwendell commented on a diff in the pull request:

https://github.com/apache/spark/pull/2922#discussion_r19325043
  
--- Diff: core/src/main/scala/org/apache/spark/broadcast/Broadcast.scala ---
@@ -87,10 +91,13 @@ abstract class Broadcast[T: ClassTag](val id: Long) 
extends Serializable {
   /**
* Destroy all data and metadata related to this broadcast variable. Use 
this with caution;
* once a broadcast variable has been destroyed, it cannot be used again.
+   * @param blocking Whether to block until destroy has completed
*/
-  private[spark] def destroy(blocking: Boolean) {
+  def destroy(blocking: Boolean) {
--- End diff --

should we only expose a version where blocking is set to true for users? It 
seems like asynchronous destroy is a bit more complex. @shivaram does your app 
need the async version?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4030] Make destroy public for broadcast...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2922#issuecomment-60350746
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22125/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3812 Build changes to publish effective ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/2921


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4030] Make destroy public for broadcast...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2922#issuecomment-60350745
  
  [Test build #22125 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22125/consoleFull)
 for   PR 2922 at commit 
[`e80c1ab`](https://github.com/apache/spark/commit/e80c1abca25f13e2a6e9b0f0b83a9fb7032ba4ca).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `abstract class Broadcast[T: ClassTag](val id: Long) extends 
Serializable with Logging `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3812 Build changes to publish effective ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2921#issuecomment-60350640
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22116/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4030] Make destroy public for broadcast...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2922#issuecomment-60350674
  
  [Test build #22125 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22125/consoleFull)
 for   PR 2922 at commit 
[`e80c1ab`](https://github.com/apache/spark/commit/e80c1abca25f13e2a6e9b0f0b83a9fb7032ba4ca).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3812 Build changes to publish effective ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2921#issuecomment-60350635
  
  [Test build #22116 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22116/consoleFull)
 for   PR 2921 at commit 
[`8841491`](https://github.com/apache/spark/commit/884149174e8bbab74de7ac99d968fe668f428c37).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2615#issuecomment-60350596
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22124/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2615#issuecomment-60350594
  
  [Test build #22124 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22124/consoleFull)
 for   PR 2615 at commit 
[`4981fa9`](https://github.com/apache/spark/commit/4981fa93ee3479af9bca1ea961ac8b85fd85cb45).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2762#issuecomment-60350429
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22118/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2762#issuecomment-60350423
  
  [Test build #22118 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22118/consoleFull)
 for   PR 2762 at commit 
[`5bb1b9d`](https://github.com/apache/spark/commit/5bb1b9d2a8ecbc50e5883365dc78624c6bf1885d).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  class DeferredObjectAdapter(oi: ObjectInspector) extends 
DeferredObject `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2615#issuecomment-60350350
  
  [Test build #22124 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22124/consoleFull)
 for   PR 2615 at commit 
[`4981fa9`](https://github.com/apache/spark/commit/4981fa93ee3479af9bca1ea961ac8b85fd85cb45).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2615#issuecomment-60350230
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22122/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4030] Make destroy public for broadcast...

[shivaram]

GitHub user shivaram opened a pull request:

https://github.com/apache/spark/pull/2922

[SPARK-4030] Make destroy public for broadcast variables

This change makes the destroy function public for broadcast variables. 
Motivation for the change is described in 
https://issues.apache.org/jira/browse/SPARK-4030.
This patch also logs where destroy was called from if a broadcast variable 
is used after destruction.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shivaram/spark-1 broadcast-destroy

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/2922.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2922


commit e80c1abca25f13e2a6e9b0f0b83a9fb7032ba4ca
Author: Shivaram Venkataraman 
Date:   2014-10-24T06:36:46Z

Make destroy public for broadcast variables
Also log where destroy was called from if a broadcast variable is used 
after destruction.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4030] Make destroy public for broadcast...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/2922#issuecomment-60350235
  
cc @pwendell @rxin for review


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

[ScrapCodes]

Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/2615#issuecomment-60350175
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2615#issuecomment-60350036
  
  [Test build #22123 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22123/consoleFull)
 for   PR 2615 at commit 
[`4981fa9`](https://github.com/apache/spark/commit/4981fa93ee3479af9bca1ea961ac8b85fd85cb45).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2615#issuecomment-60350038
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22123/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2615#issuecomment-60349835
  
  [Test build #22123 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22123/consoleFull)
 for   PR 2615 at commit 
[`4981fa9`](https://github.com/apache/spark/commit/4981fa93ee3479af9bca1ea961ac8b85fd85cb45).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/2911#issuecomment-60349709
  
Hmm, looks like that might have been a transient failure.  Just to be sure, 
though, I'm going to run this one more time to make sure that it still passes, 
then merge it (since I don't think that any issues we'll observe will be caused 
by this small snappy-java version bump).

Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4061][SQL] We cannot use EOL character ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2908#issuecomment-60349557
  
  [Test build #22121 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22121/consoleFull)
 for   PR 2908 at commit 
[`d98c99e`](https://github.com/apache/spark/commit/d98c99eec6b8e1b765e77ece8e50c40dc38ea1ef).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4061][SQL] We cannot use EOL character ...

[sarutak]

Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/2908#issuecomment-60349437
  
Thanks @liancheng ! I've refactored and added test cases.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

[ScrapCodes]

Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/2615#discussion_r19324476
  
--- Diff: bin/compute-classpath.sh ---
@@ -36,6 +34,18 @@ else
   CLASSPATH="$CLASSPATH:$FWDIR/conf"
 fi
 
+if [ -z "$SCALA_VERSION" ]; then
--- End diff --

SCALA_VERSION is here for historical reasons. @pwendell Should we change 
this to SPARK_SCALA_VERSION as part of this patch, as @retronym suggested ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19324425
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -33,6 +33,7 @@ private[streaming]
 class FileInputDStream[K: ClassTag, V: ClassTag, F <: NewInputFormat[K,V] 
: ClassTag](
 @transient ssc_ : StreamingContext,
 directory: String,
+depth: Int = 0,
--- End diff --

Would better to add a `require(depth >= 0)` in the constructor to reinforce 
the contract.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

[retronym]

Github user retronym commented on a diff in the pull request:

https://github.com/apache/spark/pull/2615#discussion_r19324410
  
--- Diff: bin/compute-classpath.sh ---
@@ -20,7 +20,7 @@
 # This script computes Spark's classpath and prints it to stdout; it's 
used by both the "run"
 # script and the ExecutorRunner in standalone cluster mode.
 
-SCALA_VERSION=2.10
+SCALA_VERSION=${SCALA_VERSION:-"2.10"}
--- End diff --

Arguably, if you are picking this up from the environment, you should call 
it `SPARK_SCALA_VERSION`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: simplify serializer, use AutoBatchedSerializer...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2920#issuecomment-60348982
  
  [Test build #22120 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22120/consoleFull)
 for   PR 2920 at commit 
[`eb3938d`](https://github.com/apache/spark/commit/eb3938d53230fc8ed73962171b947edca15b2fec).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

[ScrapCodes]

Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/2615#discussion_r19324374
  
--- Diff: conf/spark-env.sh.template ---
@@ -3,6 +3,9 @@
 # This file is sourced when running various Spark programs.
 # Copy it as spark-env.sh and edit that to configure Spark for your site.
 
+# Uncomment this if you plan to use scala 2.11
+# SCALA_VERSION=2.11
--- End diff --

Yeah, makes sense.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

[pwendell]

Github user pwendell commented on a diff in the pull request:

https://github.com/apache/spark/pull/2615#discussion_r19324296
  
--- Diff: conf/spark-env.sh.template ---
@@ -3,6 +3,9 @@
 # This file is sourced when running various Spark programs.
 # Copy it as spark-env.sh and edit that to configure Spark for your site.
 
+# Uncomment this if you plan to use scala 2.11
+# SCALA_VERSION=2.11
--- End diff --

I just don't want to put it in this because many users look at this and it 
will confuse them. We will publish separate spark binaries for Scala 2.10 and 
2.11, so most users won't ever want to set this. We can include it in developer 
documentation somewhere else.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

[pwendell]

Github user pwendell commented on a diff in the pull request:

https://github.com/apache/spark/pull/2615#discussion_r19324270
  
--- Diff: dev/change-version-to-2.10.sh ---
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+find -name 'pom.xml' -exec sed -i 's|\(artifactId.*\)_2.11|\1_2.10|g' {}  
\;
--- End diff --

Okay - fine to leave it as is then


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2911#issuecomment-60348540
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22114/consoleFull)
 for   PR 2911 at commit 
[`adec96c`](https://github.com/apache/spark/commit/adec96cde4785c542d4b89b906c56e070afd5dc9).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4050][SQL] Fix caching of temporary tab...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2912#issuecomment-60348258
  
  [Test build #417 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/417/consoleFull)
 for   PR 2912 at commit 
[`9c822d4`](https://github.com/apache/spark/commit/9c822d43f4906747090d04ab04e5a4dcd0bb61e5).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3907][sql] add truncate table support

[marmbrus]

Github user marmbrus commented on a diff in the pull request:

https://github.com/apache/spark/pull/2770#discussion_r19324134
  
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -121,7 +121,8 @@ private[hive] object HiveQl {
   // Commands that we do not need to explain.
   protected val noExplainCommands = Seq(
 "TOK_CREATETABLE",
-"TOK_DESCTABLE"
+"TOK_DESCTABLE",
+"TOK_TRUNCATETABLE"
--- End diff --

Can you add a comment below explaining this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3822] Executor scaling mechanism for Ya...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2840#issuecomment-60348130
  
  [Test build #22119 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22119/consoleFull)
 for   PR 2840 at commit 
[`005a124`](https://github.com/apache/spark/commit/005a124fc7c2637c4b808cd4bfcd094d4f31176f).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19324027
  
--- Diff: 
streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala ---
@@ -141,6 +141,108 @@ class InputStreamsSuite extends TestSuiteBase with 
BeforeAndAfter {
 conf.set("spark.streaming.clock", 
"org.apache.spark.streaming.util.ManualClock")
   }
 
+  test("file input stream -depth = 1") {
--- End diff --

Space after `-`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19324022
  
--- Diff: 
streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala ---
@@ -91,7 +91,7 @@ class InputStreamsSuite extends TestSuiteBase with 
BeforeAndAfter {
   }
 
 
-  test("file input stream") {
+  test("file input stream -depth = 0  ") {
--- End diff --

Space after `=`, and remove the trailing spaces in the string.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3822] Executor scaling mechanism for Ya...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/2840#issuecomment-60347788
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19324008
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -230,16 +266,37 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: 
NewInputFormat[K,V] : Clas
 if (minNewFileModTime < 0 || modTime < minNewFileModTime) {
   minNewFileModTime = modTime
 }
+if(path.getName().startsWith("_")){
--- End diff --

Space after `if` and before `{`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323999
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -118,6 +120,37 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: 
NewInputFormat[K,V] : Clas
 (newFiles, filter.minNewFileModTime)
   }
 
+  def getPathList(path:Path, fs:FileSystem):List[Path]={
+var pathList = List[Path]()
+pathList = path:: pathList
+var tmp =List[Path]()
+tmp=path::tmp
+for(i <- 0 until depth){
+  tmp =getSubPathList(tmp,fs)
+  pathList=tmp:::pathList
+}
+pathList.filter(path=>{
+  val  modTime = fs.getFileStatus(path).getModificationTime
+  logDebug(s"Mod time for $path is $modTime")
+  if (modTime > ignoreTime) {
+logDebug(s"Mod time $modTime more than ignore time $ignoreTime")
+true
+  }
+  else false
+})
+  }
+
+  def getSubPathList(path:List[Path],fs:FileSystem):List[Path]={
+val filter = new SubPathFilter()
+var pathList = List[Path]()
+path.map(subPath=>{
+ fs.listStatus(subPath,filter).map(x=>{
+pathList = x.getPath()::pathList
--- End diff --

Spaces round `::`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323995
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -118,6 +120,37 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: 
NewInputFormat[K,V] : Clas
 (newFiles, filter.minNewFileModTime)
   }
 
+  def getPathList(path:Path, fs:FileSystem):List[Path]={
+var pathList = List[Path]()
+pathList = path:: pathList
+var tmp =List[Path]()
+tmp=path::tmp
+for(i <- 0 until depth){
+  tmp =getSubPathList(tmp,fs)
+  pathList=tmp:::pathList
+}
+pathList.filter(path=>{
+  val  modTime = fs.getFileStatus(path).getModificationTime
+  logDebug(s"Mod time for $path is $modTime")
+  if (modTime > ignoreTime) {
+logDebug(s"Mod time $modTime more than ignore time $ignoreTime")
+true
+  }
+  else false
+})
+  }
+
+  def getSubPathList(path:List[Path],fs:FileSystem):List[Path]={
+val filter = new SubPathFilter()
+var pathList = List[Path]()
+path.map(subPath=>{
+ fs.listStatus(subPath,filter).map(x=>{
--- End diff --

Similar to [this 
one](https://github.com/apache/spark/pull/2765/files#r19323955). And space 
after `,`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323992
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -118,6 +120,37 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: 
NewInputFormat[K,V] : Clas
 (newFiles, filter.minNewFileModTime)
   }
 
+  def getPathList(path:Path, fs:FileSystem):List[Path]={
+var pathList = List[Path]()
+pathList = path:: pathList
+var tmp =List[Path]()
+tmp=path::tmp
+for(i <- 0 until depth){
+  tmp =getSubPathList(tmp,fs)
+  pathList=tmp:::pathList
+}
+pathList.filter(path=>{
+  val  modTime = fs.getFileStatus(path).getModificationTime
+  logDebug(s"Mod time for $path is $modTime")
+  if (modTime > ignoreTime) {
+logDebug(s"Mod time $modTime more than ignore time $ignoreTime")
+true
+  }
+  else false
+})
+  }
+
+  def getSubPathList(path:List[Path],fs:FileSystem):List[Path]={
+val filter = new SubPathFilter()
+var pathList = List[Path]()
+path.map(subPath=>{
--- End diff --

Similar to [this 
one](https://github.com/apache/spark/pull/2765/files#r19323955).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323986
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -118,6 +120,37 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: 
NewInputFormat[K,V] : Clas
 (newFiles, filter.minNewFileModTime)
   }
 
+  def getPathList(path:Path, fs:FileSystem):List[Path]={
+var pathList = List[Path]()
+pathList = path:: pathList
+var tmp =List[Path]()
+tmp=path::tmp
+for(i <- 0 until depth){
+  tmp =getSubPathList(tmp,fs)
+  pathList=tmp:::pathList
+}
+pathList.filter(path=>{
+  val  modTime = fs.getFileStatus(path).getModificationTime
+  logDebug(s"Mod time for $path is $modTime")
+  if (modTime > ignoreTime) {
+logDebug(s"Mod time $modTime more than ignore time $ignoreTime")
+true
+  }
+  else false
+})
+  }
+
+  def getSubPathList(path:List[Path],fs:FileSystem):List[Path]={
--- End diff --

Spaces after `:` and `,`, and around `=`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323975
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -118,6 +120,37 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: 
NewInputFormat[K,V] : Clas
 (newFiles, filter.minNewFileModTime)
   }
 
+  def getPathList(path:Path, fs:FileSystem):List[Path]={
+var pathList = List[Path]()
+pathList = path:: pathList
+var tmp =List[Path]()
+tmp=path::tmp
+for(i <- 0 until depth){
+  tmp =getSubPathList(tmp,fs)
+  pathList=tmp:::pathList
+}
+pathList.filter(path=>{
+  val  modTime = fs.getFileStatus(path).getModificationTime
+  logDebug(s"Mod time for $path is $modTime")
+  if (modTime > ignoreTime) {
+logDebug(s"Mod time $modTime more than ignore time $ignoreTime")
+true
+  }
+  else false
--- End diff --

Please reformat this `if` according to the coding convention, like this:

```scala
if (...) {
  ...
} else {
  ...
}
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323955
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -118,6 +120,37 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: 
NewInputFormat[K,V] : Clas
 (newFiles, filter.minNewFileModTime)
   }
 
+  def getPathList(path:Path, fs:FileSystem):List[Path]={
+var pathList = List[Path]()
+pathList = path:: pathList
+var tmp =List[Path]()
+tmp=path::tmp
+for(i <- 0 until depth){
+  tmp =getSubPathList(tmp,fs)
+  pathList=tmp:::pathList
+}
+pathList.filter(path=>{
--- End diff --

Spaces before `=>`, and remove the redundant braces, like this:

```scala
pathList.filter { path =>
  ...
}
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323941
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -118,6 +120,37 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: 
NewInputFormat[K,V] : Clas
 (newFiles, filter.minNewFileModTime)
   }
 
+  def getPathList(path:Path, fs:FileSystem):List[Path]={
+var pathList = List[Path]()
+pathList = path:: pathList
+var tmp =List[Path]()
+tmp=path::tmp
+for(i <- 0 until depth){
+  tmp =getSubPathList(tmp,fs)
+  pathList=tmp:::pathList
--- End diff --

Spaces around `=` and `:::`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323932
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -118,6 +120,37 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: 
NewInputFormat[K,V] : Clas
 (newFiles, filter.minNewFileModTime)
   }
 
+  def getPathList(path:Path, fs:FileSystem):List[Path]={
+var pathList = List[Path]()
+pathList = path:: pathList
+var tmp =List[Path]()
+tmp=path::tmp
+for(i <- 0 until depth){
--- End diff --

Space before `{`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323936
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -118,6 +120,37 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: 
NewInputFormat[K,V] : Clas
 (newFiles, filter.minNewFileModTime)
   }
 
+  def getPathList(path:Path, fs:FileSystem):List[Path]={
+var pathList = List[Path]()
+pathList = path:: pathList
+var tmp =List[Path]()
+tmp=path::tmp
+for(i <- 0 until depth){
+  tmp =getSubPathList(tmp,fs)
--- End diff --

Space after `=` and `,`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2762#issuecomment-60347589
  
  [Test build #22118 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22118/consoleFull)
 for   PR 2762 at commit 
[`5bb1b9d`](https://github.com/apache/spark/commit/5bb1b9d2a8ecbc50e5883365dc78624c6bf1885d).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323929
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -118,6 +120,37 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: 
NewInputFormat[K,V] : Clas
 (newFiles, filter.minNewFileModTime)
   }
 
+  def getPathList(path:Path, fs:FileSystem):List[Path]={
+var pathList = List[Path]()
+pathList = path:: pathList
+var tmp =List[Path]()
+tmp=path::tmp
--- End diff --

Spaces around `=` and `::`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323922
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -118,6 +120,37 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: 
NewInputFormat[K,V] : Clas
 (newFiles, filter.minNewFileModTime)
   }
 
+  def getPathList(path:Path, fs:FileSystem):List[Path]={
+var pathList = List[Path]()
+pathList = path:: pathList
+var tmp =List[Path]()
--- End diff --

Space after `=`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323915
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -118,6 +120,37 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: 
NewInputFormat[K,V] : Clas
 (newFiles, filter.minNewFileModTime)
   }
 
+  def getPathList(path:Path, fs:FileSystem):List[Path]={
--- End diff --

Space after all `:`s.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323896
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -351,8 +351,8 @@ class StreamingContext private[streaming] (
 K: ClassTag,
 V: ClassTag,
 F <: NewInputFormat[K, V]: ClassTag
-  ] (directory: String, filter: Path => Boolean, newFilesOnly: Boolean): 
InputDStream[(K, V)] = {
-new FileInputDStream[K, V, F](this, directory, filter, newFilesOnly)
+  ] (directory: String,filter: Path => Boolean, newFilesOnly: Boolean, 
depth : Int =0): InputDStream[(K, V)] = {
+new FileInputDStream[K, V, F](this, directory,depth, filter, 
newFilesOnly)
--- End diff --

Space before `depth`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323911
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -104,7 +105,8 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: 
NewInputFormat[K,V] : Clas
 logDebug("Trying to get new files for time " + currentTime)
 lastNewFileFindingTime = System.currentTimeMillis
 val filter = new CustomPathFilter(currentTime)
-val newFiles = fs.listStatus(directoryPath, 
filter).map(_.getPath.toString)
+val pathList = getPathList(directoryPath,fs)
+val newFiles = 
fs.listStatus(pathList.toArray,filter).map(_.getPath.toString)
--- End diff --

Space after `,`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323906
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
 ---
@@ -104,7 +105,8 @@ class FileInputDStream[K: ClassTag, V: ClassTag, F <: 
NewInputFormat[K,V] : Clas
 logDebug("Trying to get new files for time " + currentTime)
 lastNewFileFindingTime = System.currentTimeMillis
 val filter = new CustomPathFilter(currentTime)
-val newFiles = fs.listStatus(directoryPath, 
filter).map(_.getPath.toString)
+val pathList = getPathList(directoryPath,fs)
--- End diff --

Space after `,`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323904
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -363,8 +363,8 @@ class StreamingContext private[streaming] (
* file system. File names starting with . are ignored.
* @param directory HDFS directory to monitor for new file
*/
-  def textFileStream(directory: String): DStream[String] = {
-fileStream[LongWritable, Text, 
TextInputFormat](directory).map(_._2.toString)
+  def textFileStream(directory: String,depth: Int =0): DStream[String] = {
+fileStream[LongWritable, Text, 
TextInputFormat](directory,depth).map(_._2.toString)
--- End diff --

Space after `,`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323902
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -363,8 +363,8 @@ class StreamingContext private[streaming] (
* file system. File names starting with . are ignored.
* @param directory HDFS directory to monitor for new file
*/
-  def textFileStream(directory: String): DStream[String] = {
-fileStream[LongWritable, Text, 
TextInputFormat](directory).map(_._2.toString)
+  def textFileStream(directory: String,depth: Int =0): DStream[String] = {
--- End diff --

Space after `,` and `=`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323893
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -351,8 +351,8 @@ class StreamingContext private[streaming] (
 K: ClassTag,
 V: ClassTag,
 F <: NewInputFormat[K, V]: ClassTag
-  ] (directory: String, filter: Path => Boolean, newFilesOnly: Boolean): 
InputDStream[(K, V)] = {
-new FileInputDStream[K, V, F](this, directory, filter, newFilesOnly)
+  ] (directory: String,filter: Path => Boolean, newFilesOnly: Boolean, 
depth : Int =0): InputDStream[(K, V)] = {
--- End diff --

- Remove space before `:`
- Add space after `=`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323886
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -331,8 +331,8 @@ class StreamingContext private[streaming] (
 K: ClassTag,
 V: ClassTag,
 F <: NewInputFormat[K, V]: ClassTag
-  ] (directory: String): InputDStream[(K, V)] = {
-new FileInputDStream[K, V, F](this, directory)
+  ] (directory: String,depth :Int =0): InputDStream[(K, V)] = {
+new FileInputDStream[K, V, F](this, directory,depth)
--- End diff --

Space after `,`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2765#discussion_r19323885
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala ---
@@ -331,8 +331,8 @@ class StreamingContext private[streaming] (
 K: ClassTag,
 V: ClassTag,
 F <: NewInputFormat[K, V]: ClassTag
-  ] (directory: String): InputDStream[(K, V)] = {
-new FileInputDStream[K, V, F](this, directory)
+  ] (directory: String,depth :Int =0): InputDStream[(K, V)] = {
--- End diff --

- Add space after `,`
- Remove space before `:`
- Add space after `:`
- Add space after `=`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: simplify serializer, use AutoBatchedSerializer...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2920#issuecomment-60347144
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22115/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/2762#issuecomment-60347169
  
test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4000][BUILD] Sends archived unit tests ...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/2845#discussion_r19323760
  
--- Diff: dev/run-tests-jenkins ---
@@ -92,12 +92,39 @@ function post_message () {
   echo " > api_response: ${api_response}" >&2
   echo " > data: ${data}" >&2
   fi
-  
+
   if [ "$curl_status" -eq 0 ] && [ "$http_code" -eq "201" ]; then
 echo " > Post successful."
   fi
 }
 
+function send_archived_logs () {
+  echo "Archiving unit tests logs..."
+
+  local log_files=$(find . -name "unit-tests.log")
+
+  if [ -z "$log_files" ]; then
+echo "> No log files found." >&2
+  else
+local log_archive="unit-tests-logs.tar.gz"
+echo "$log_files" | xargs tar czf ${log_archive}
+
+local 
jenkins_build_dir=${JENKINS_HOME}/jobs/${JOB_NAME}/builds/${BUILD_NUMBER}
+local scp_output=$(scp ${log_archive} 
amp-jenkins-master:${jenkins_build_dir}/${log_archive})
--- End diff --

Cool, thanks :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3812 Build changes to publish effective ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2921#issuecomment-60346730
  
  [Test build #22116 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22116/consoleFull)
 for   PR 2921 at commit 
[`8841491`](https://github.com/apache/spark/commit/884149174e8bbab74de7ac99d968fe668f428c37).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: simplify serializer, use AutoBatchedSerializer...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2920#issuecomment-60346745
  
  [Test build #22117 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22117/consoleFull)
 for   PR 2920 at commit 
[`8d77ef2`](https://github.com/apache/spark/commit/8d77ef20276af479663725ab6dd70c93e9973bb4).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3812 Build changes to publish effective ...

[ScrapCodes]

GitHub user ScrapCodes opened a pull request:

https://github.com/apache/spark/pull/2921

SPARK-3812 Build changes to publish effective pom.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ScrapCodes/spark-1 build-changes-effective-pom

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/2921.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2921


commit 553d96ba253fa98a97c61a38a33d5990a9f46d63
Author: Prashant Sharma 
Date:   2014-10-06T10:22:25Z

Shaded some unused class of an unused dep, to generate effective pom(s)

commit 28f891ef216c0a3e99bc12649fe7f2c6b12751bf
Author: Prashant Sharma 
Date:   2014-10-21T07:24:04Z

Added a useless dependency, so that we can shade it. And realized fake 
shading works for us.

commit 0300dac5d891a5313c25d73400ccb019dd0f5a19
Author: Prashant Sharma 
Date:   2014-10-21T09:19:02Z

improved comment messages..

commit aa7b91d30f2c9e6b8475466381a3e4dd0b8b2dd2
Author: Prashant Sharma 
Date:   2014-10-22T09:04:55Z

used an unused dep.

commit 884149174e8bbab74de7ac99d968fe668f428c37
Author: Prashant Sharma 
Date:   2014-10-24T05:32:12Z

Fixed broken maven build.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: simplify serializer, use AutoBatchedSerializer...

[davies]

GitHub user davies opened a pull request:

https://github.com/apache/spark/pull/2920

simplify serializer, use AutoBatchedSerializer by default.

This PR simplify serializer, always use batched serializer 
(AutoBatchedSerializer as default), even batch size is 1.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davies/spark fix_autobatch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/2920.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2920


commit 3178077bad940912780d4024d522a65746881630
Author: Davies Liu 
Date:   2014-10-24T05:27:37Z

simplify serializer, use AutoBatchedSerializer by default.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2713] Executors of same application in ...

[li-zhihui]

Github user li-zhihui commented on the pull request:

https://github.com/apache/spark/pull/1616#issuecomment-60346005
  
@andrewor14 I guess the failure is non-interrelated with the patch. But I 
don't know why failed again, can you give me some advise? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4065] Add check for IPython on Windows

[sarutak]

Github user sarutak commented on a diff in the pull request:

https://github.com/apache/spark/pull/2910#discussion_r19323459
  
--- Diff: bin/pyspark2.cmd ---
@@ -33,7 +33,7 @@ for %%d in 
("%FWDIR%assembly\target\scala-%SCALA_VERSION%\spark-assembly*hadoop*
 )
 if [%FOUND_JAR%] == [0] (
   echo Failed to find Spark assembly JAR.
-  echo You need to build Spark before running this program.
+  echo You need to build Spark with sbt\sbt assembly before running this 
program.
--- End diff --

Removing "with sbt\sbt assembly" is correct. see #2640.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4000][BUILD] Sends archived unit tests ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/2845


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4000][BUILD] Sends archived unit tests ...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/2845#issuecomment-60345496
  
Alright, I'm going to merge this now and I'll hotfix the 
`JENKINS_MASTER_HOSTNAME` myself later.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4000][BUILD] Sends archived unit tests ...

[pwendell]

Github user pwendell commented on the pull request:

https://github.com/apache/spark/pull/2845#issuecomment-60345439
  
Merge away!

On Thu, Oct 23, 2014 at 10:10 PM, Josh Rosen 
wrote:

> Alright, I've modified the pull request builder job to inject the master
> hostname as JENKINS_MASTER_HOSTNAME.
>
> —
> Reply to this email directly or view it on GitHub
> .
>


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4000][BUILD] Sends archived unit tests ...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/2845#issuecomment-60345312
  
Alright, I've modified the pull request builder job to inject the master 
hostname as `JENKINS_MASTER_HOSTNAME`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3223 runAsSparkUser cannot change HDFS w...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2126#issuecomment-60345207
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22113/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3572] [sql] [mllib] User-Defined Types ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2919#issuecomment-60345200
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22112/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3572] [sql] [mllib] User-Defined Types ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2919#issuecomment-60345199
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22112/consoleFull)
 for   PR 2919 at commit 
[`3de3d76`](https://github.com/apache/spark/commit/3de3d768951f020bf0876da2f40cc098210fcf05).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  case class Params(`
  * `//   in some cases, such as when a class is enclosed in an 
object (in which case`
  * `abstract class UserDefinedType[UserType] extends DataType with 
Serializable `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-3223 runAsSparkUser cannot change HDFS w...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2126#issuecomment-60345204
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22113/consoleFull)
 for   PR 2126 at commit 
[`ea7e4cd`](https://github.com/apache/spark/commit/ea7e4cdca4666f958acb68aae0c88cf1e32f9481).
 * This patch **fails some tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4000][BUILD] Sends archived unit tests ...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/2845#discussion_r19323244
  
--- Diff: dev/run-tests-jenkins ---
@@ -92,12 +92,39 @@ function post_message () {
   echo " > api_response: ${api_response}" >&2
   echo " > data: ${data}" >&2
   fi
-  
+
   if [ "$curl_status" -eq 0 ] && [ "$http_code" -eq "201" ]; then
 echo " > Post successful."
   fi
 }
 
+function send_archived_logs () {
+  echo "Archiving unit tests logs..."
+
+  local log_files=$(find . -name "unit-tests.log")
+
+  if [ -z "$log_files" ]; then
+echo "> No log files found." >&2
+  else
+local log_archive="unit-tests-logs.tar.gz"
+echo "$log_files" | xargs tar czf ${log_archive}
+
+local 
jenkins_build_dir=${JENKINS_HOME}/jobs/${JOB_NAME}/builds/${BUILD_NUMBER}
+local scp_output=$(scp ${log_archive} 
amp-jenkins-master:${jenkins_build_dir}/${log_archive})
--- End diff --

@liancheng I should be able to modify our pull request builders to inject 
this variable as `JENKINS_MASTER_HOSTNAME`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4000][BUILD] Sends archived unit tests ...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/2845#issuecomment-60345019
  
This looks good to me.  @pwendell, any more feedback?  I'd like to get this 
merged soon in order to help debug a few Jenkins failures.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/2911#issuecomment-60344962
  
Actually, I don't think that "deflake build" plugin will necessarily work 
as expected given all of the customization in our build; I guess it was added 
for another project that shares the Jenkins server with us.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/2911#issuecomment-60344903
  
@xerial Here's a link to the exception from that most recent test failure:


https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22106/testReport/junit/org.apache.spark.util.collection/ExternalAppendOnlyMapSuite/spilling_with_compression/

In case that link breaks, here's the driver stacktrace:

```
sbt.ForkMain$ForkError: Test failed with compression using codec 
org.apache.spark.io.SnappyCompressionCodec:

Job aborted due to stage failure: Task 0 in stage 4.0 failed 4 times, most 
recent failure: Lost task 0.3 in stage 4.0 (TID 14, localhost): 
java.io.IOException: unexpected exception type

java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1538)

java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1025)

java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)

java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)

java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)

java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)

java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)

org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)

org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)

org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:163)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at 
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1191)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1180)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1179)
at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at 
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1179)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:694)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:694)
at scala.Option.foreach(Option.scala:236)
at 
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:694)
at 
org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1397)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at 
org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1352)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
```

This isn't the full stacktrace of the actual error, which took place in an 
executor. Until we merge #2845, I don't think that I'll have an easy way to 
grab the full executor logs from Jenkins. 

I wasn't able to reproduce this failure locally.  I'm going to try our new 
experimental "Deflake build" Jenkins button, which reruns only the failing 
tests, in order to see if I can reproduce this.  If so, I'll SSH in and grab 
the full logs.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

--

[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2911#issuecomment-60344869
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22114/consoleFull)
 for   PR 2911 at commit 
[`adec96c`](https://github.com/apache/spark/commit/adec96cde4785c542d4b89b906c56e070afd5dc9).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3822] Executor scaling mechanism for Ya...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2840#issuecomment-60344082
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22111/consoleFull)
 for   PR 2840 at commit 
[`005a124`](https://github.com/apache/spark/commit/005a124fc7c2637c4b808cd4bfcd094d4f31176f).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  case class AddWebUIFilter(filterName:String, filterParams: 
Map[String, String], proxyBase: String)`
  * `  case class RequestExecutors(numExecutors: Int) extends 
CoarseGrainedClusterMessage`
  * `  case class KillExecutors(executorIds: Seq[String]) extends 
CoarseGrainedClusterMessage`
  * `class CoarseGrainedSchedulerBackend(scheduler: TaskSchedulerImpl, val 
actorSystem: ActorSystem)`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3822] Executor scaling mechanism for Ya...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2840#issuecomment-60344084
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22111/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4070] [WIP] Add WebUITableBuilder to si...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2852#issuecomment-60343806
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22107/consoleFull)
 for   PR 2852 at commit 
[`1975cd6`](https://github.com/apache/spark/commit/1975cd6364d6403ec2d7f885743020b55b10fea9).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class UITableColumn[T, V](`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4070] [WIP] Add WebUITableBuilder to si...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2852#issuecomment-60343809
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22107/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...

[chenghao-intel]

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/2762#issuecomment-60343740
  
test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2663] [SQL] Support the Grouping Set

[chenghao-intel]

Github user chenghao-intel commented on the pull request:

https://github.com/apache/spark/pull/1567#issuecomment-60343575
  
@rxin @marmbrus , I've uploaded an draft design doc in jira. 
https://issues.apache.org/jira/secure/attachment/12676811/grouping_set.pdf, 
sorry it maybe not cover every detail, let me know if you have any confusion. 

@marmbrus 
>The creation of bit vectors seems like a very implementation focused 
physical concern. I'm curious if this could be restricted to the actual 
physical operator.

Yeah, It's very reasonable, I was thinking of this either. 
However, the bit vectors stuff don't rely on physical execution engine, and 
it's slightly different with the Aggregate, which has the optimization of 
mapside aggregation for spark execution.

Besides, the attribute reference pass down to the parent logical operator 
need to be correctly set in logical plan analyzing. 

Anyway, I will consider your suggestion, after all, we should keep the 
Logical Plan for "describing what to do", not "how to do".

>Adding a new type of attribute reference for virtual columns might be a 
lot of overhead. Is this really necessary?

A concrete `VirtualColumn` instance is very helpful in attribute 
referencing, and pattern matching, probably better than a name convention. 
Sorry, maybe I didn't understand your mean, we can discuss that in the code 
review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5

[xerial]

Github user xerial commented on the pull request:

https://github.com/apache/spark/pull/2911#issuecomment-60343444
  
@JoshRosen If you have the stack trace of this error, please let me know. I 
would like to check it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2713] Executors of same application in ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/1616#issuecomment-60343237
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22108/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2713] Executors of same application in ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1616#issuecomment-60343235
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22108/consoleFull)
 for   PR 1616 at commit 
[`36940df`](https://github.com/apache/spark/commit/36940df56f2af4071bf903559be8ae64d82a1808).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2746#issuecomment-60343190
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22105/consoleFull)
 for   PR 2746 at commit 
[`092d1fd`](https://github.com/apache/spark/commit/092d1fdc9f8f450a447a3c8292a78eef9937ab3a).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP][SPARK-3795] Heuristics for dynamically s...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2746#issuecomment-60343192
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22105/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

[ScrapCodes]

Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/2615#discussion_r19322483
  
--- Diff: dev/change-version-to-2.11.sh ---
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
--- End diff --

This was not possible.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [ SPARK-1812] Adjust build system and tests to...

[ScrapCodes]

Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/2615#discussion_r19322477
  
--- Diff: conf/spark-env.sh.template ---
@@ -3,6 +3,9 @@
 # This file is sourced when running various Spark programs.
 # Copy it as spark-env.sh and edit that to configure Spark for your site.
 
+# Uncomment this if you plan to use scala 2.11
+# SCALA_VERSION=2.11
--- End diff --

Is it okay to leave this as is, even if we have automatic guessing of scala 
version on ? Otherwise there is no way to know that one can set a preferred 
scala version too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org