[GitHub] spark pull request: [SPARK-11265] [YARN] YarnClient can't get toke...

[steveloughran]

Github user steveloughran commented on the pull request:

https://github.com/apache/spark/pull/9232#issuecomment-152815974
  
Will do. Same JIRA or a new backport one?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11305] [DOCS] Remove Third-Party Hadoop...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/9298#issuecomment-152822556
  
Merged to master for 1.6


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11389][WIP][CORE] Add support for off-h...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/9344#issuecomment-152802034
  
Looking a bit more closely, it looks like all existing implementations of 
`MemoryConsumer.spill()` will only end up reporting the size of Tugnsten pages 
that are spilled. If Tungsten pages are allocated in-heap, then it makes sense 
to try to spill in response to requests heap memory. If we're in off-heap mode, 
though, then it doesn't make sense to try to spill when we're running short in 
on-heap memory; in that mode, we should only spill in response to failed 
off-heap memory requests.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11338][WebUI] Prepend app links on Hist...

[ckadner]

Github user ckadner commented on the pull request:

https://github.com/apache/spark/pull/9291#issuecomment-152799657
  
@vanzin -- I pushed the nit pickity fixes :mag:  -- Thanks :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11195][CORE] Use correct classloader fo...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/9367#issuecomment-152822504
  
@choochootrain are you able to put together a small test like what @yhuai 
mentions? then I think this is good to go.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-11344: Made ApplicationDescription and D...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/9299#issuecomment-152822457
  
@jacek-lewandowski on one final re-read my last question is -- why does 
`ApplicationDescription` still have `appUiUrl` then? in several cases it's just 
copied into an `ApplicationInfo` that contains it. You have an 'original' value 
in the description and some possibly changed copy in the info object. Is that 
the intent? can it move out of the description class entirely?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11338][WebUI] Prepend app links on Hist...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9291#issuecomment-152797984
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9162][SQL] Implement code generation fo...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/9270#issuecomment-152802120
  
I'm traveling right now. Will take a look when I get back.

cc @davies also.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11424] Guard against double-close() of ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9388#issuecomment-152803978
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11314] [YARN] add service API and test ...

[steveloughran]

Github user steveloughran commented on the pull request:

https://github.com/apache/spark/pull/9182#issuecomment-152816395
  
and I've already gone and moved to strings. never mind.

the existing attempt IDs are nice for humans in the web UI, and potentially 
in the rest, but don't let you hook up to yarn's internals.

now, should I stay with String or roll back?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11305] [DOCS] Remove Third-Party Hadoop...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/9298


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11389][WIP][CORE] Add support for off-h...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/9344#issuecomment-152800152
  
One bad complication: until we can _completely_ support off-heap memory for 
execution, we need to perform separate accounting for on-heap and off-heap 
memory, so `spill()` will need to report accurate information on the amount of 
on-heap and off-heap memory that's freed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...

[sun-rui]

Github user sun-rui commented on the pull request:

https://github.com/apache/spark/pull/9218#issuecomment-152797513
  
yes, I agree:  throw a complex types not supported error message


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11413][BUILD] Bump joda-time version to...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/9379#issuecomment-152822945
  
Seems OK to me in that the release notes do not indicate any incompatible 
changes, and the tests pass, and it fixes some bugs. We don't actually use Joda 
time in Spark; it appears to be added for Hive. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11265] [YARN] YarnClient can't get toke...

[tedyu]

Github user tedyu commented on a diff in the pull request:

https://github.com/apache/spark/pull/9232#discussion_r43583071
  
--- Diff: 
yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala ---
@@ -142,6 +145,76 @@ class YarnSparkHadoopUtil extends SparkHadoopUtil {
 val containerIdString = 
System.getenv(ApplicationConstants.Environment.CONTAINER_ID.name())
 ConverterUtils.toContainerId(containerIdString)
   }
+
+  /**
+   * Obtains token for the Hive metastore, using the current user as the 
principal.
+   * Some exceptions are caught and downgraded to a log message.
+   * @param conf hadoop configuration; the Hive configuration will be 
based on this
+   * @return a token, or `None` if there's no need for a token (no 
metastore URI or principal
+   * in the config), or if a binding exception was caught and 
downgraded.
+   */
+  def obtainTokenForHiveMetastore(conf: Configuration): 
Option[Token[DelegationTokenIdentifier]] = {
+try {
+  obtainTokenForHiveMetastoreInner(conf, 
UserGroupInformation.getCurrentUser().getUserName)
+} catch {
+  case e: ClassNotFoundException =>
+logInfo(s"Hive class not found $e")
+logDebug("Hive class not found", e)
--- End diff --

Why double log the message ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11236][CORE] Update Tachyon dependency ...

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/9395#issuecomment-152833844
  
btw, the profiles that we used in hadoop 1 tests are `-Phadoop-1 
-Dhadoop.version=1.2.1 -Pkinesis-asl -Phive-thriftserver -Phive`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11236][CORE] Update Tachyon dependency ...

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/9395#issuecomment-152833780
  
cc @pwendell @srowen This is the new pr that upgrades Tachyon. I reverted 
the original one, which caused hadoop 1 test failures.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Reduce numSlices for local metrics test of Spa...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9384#issuecomment-152844736
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Reduce numSlices for local metrics test of Spa...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9384#issuecomment-152844739
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44757/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Reduce numSlices for local metrics test of Spa...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9384#issuecomment-152844573
  
**[Test build #44757 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44757/consoleFull)**
 for PR 9384 at commit 
[`eb2d1b5`](https://github.com/apache/spark/commit/eb2d1b509fde34b106c10a01e91123e7fc6fd2e9).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Reduce numSlices for local metrics test of Spa...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9384#issuecomment-152846620
  
**[Test build #44759 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44759/consoleFull)**
 for PR 9384 at commit 
[`7cc64f3`](https://github.com/apache/spark/commit/7cc64f3a5f86b371c6d2eb376e0fe4735306d9ea).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11112] DAG visualization: display RDD c...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9398#issuecomment-152848394
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10280][MLlib][PySpark][Docs] Add @since...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8690#issuecomment-152848385
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11112] DAG visualization: display RDD c...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9398#issuecomment-152848384
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10280][MLlib][PySpark][Docs] Add @since...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8690#issuecomment-152849693
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44763/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10280][MLlib][PySpark][Docs] Add @since...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8690#issuecomment-152849703
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44762/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10280][MLlib][PySpark][Docs] Add @since...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8690#issuecomment-152849692
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11440][CORE][STREAMING][BUILD] Declare ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9396#issuecomment-152850665
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11440][CORE][STREAMING][BUILD] Declare ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9396#issuecomment-152850628
  
**[Test build #44758 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44758/consoleFull)**
 for PR 9396 at commit 
[`2439d71`](https://github.com/apache/spark/commit/2439d7105022aa67a635498fa2d5145a134150d7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * `
logInfo(s\"Hive class not found $e\")`\n  * `logDebug(\"Hive class 
not found\", e)`\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11440][CORE][STREAMING][BUILD] Declare ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9396#issuecomment-152850666
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44758/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11440][CORE][STREAMING][BUILD] Declare ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9396#issuecomment-152837849
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11440][CORE][STREAMING][BUILD] Declare ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9396#issuecomment-152839904
  
**[Test build #44758 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44758/consoleFull)**
 for PR 9396 at commit 
[`2439d71`](https://github.com/apache/spark/commit/2439d7105022aa67a635498fa2d5145a134150d7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11236][CORE] Update Tachyon dependency ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9395#issuecomment-152841909
  
**[Test build #44756 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44756/consoleFull)**
 for PR 9395 at commit 
[`6802ecd`](https://github.com/apache/spark/commit/6802ecd5c957a64bfcb11107ab8390a64a0d4ad6).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * `
logInfo(s\"Hive class not found $e\")`\n  * `logDebug(\"Hive class 
not found\", e)`\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10500][SPARKR][WIP] sparkr.zip cannot b...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9390#issuecomment-152841955
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10500][SPARKR][WIP] sparkr.zip cannot b...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9390#issuecomment-152841911
  
**[Test build #44753 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44753/consoleFull)**
 for PR 9390 at commit 
[`3fbb2db`](https://github.com/apache/spark/commit/3fbb2dbb3a2f7a1936b12cd5af7eab2f31d625e7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Reduce numSlices for local metrics test of Spa...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9384#issuecomment-152846476
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Reduce numSlices for local metrics test of Spa...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9384#issuecomment-152846483
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11112] DAG visualization: display RDD c...

[andrewor14]

GitHub user andrewor14 opened a pull request:

https://github.com/apache/spark/pull/9398

[SPARK-2] DAG visualization: display RDD callsite

https://cloud.githubusercontent.com/assets/2133137/10870343/2a8cd070-807d-11e5-857a-4ebcace77b5b.png;>
@mateiz @sarutak


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrewor14/spark rdd-callsite

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9398.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9398


commit 2401599878e0b734ebeca6b57c27ec5b8a53d1ca
Author: Andrew Or 
Date:   2015-11-01T09:00:11Z

Add RDD callsite to listener

commit 61baf102c727999b3a196f4d5198d06156a0037a
Author: Andrew Or 
Date:   2015-11-01T09:07:46Z

Add RDD callsite to DAG visualization

commit 5b6c70e804fe81ca127515f8e27cc05e99771a9b
Author: Andrew Or 
Date:   2015-11-01T17:43:07Z

Merge branch 'master' of github.com:apache/spark into rdd-callsite




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10280][MLlib][PySpark][Docs] Add @since...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8690#issuecomment-152848393
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10280][MLlib][PySpark][Docs] Add @since...

[yu-iskw]

Github user yu-iskw commented on the pull request:

https://github.com/apache/spark/pull/8690#issuecomment-152848369
  
@noel-smith thank you for the review. I rebase this PR with the mater and 
then updated two parts.

- Add `@since` tags to `BGTClassifier` class and object
- Add `@since` tags to `TreeClassifierParams`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11389][WIP][CORE] Add support for off-h...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9344#issuecomment-152850333
  
**[Test build #44755 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44755/consoleFull)**
 for PR 9344 at commit 
[`b59dab9`](https://github.com/apache/spark/commit/b59dab960d6081aaa8d558d9d563ddcb445b9e4d).
 * This patch **fails from timeout after a configured wait of \`250m\`**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:\n  * 
`class ExecutionMemoryPool(memoryManager: Object, poolName: String) extends 
MemoryPool with Logging `\n  * `abstract class MemoryPool `\n  * `class 
StorageMemoryPool extends MemoryPool with Logging `\n


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11389][WIP][CORE] Add support for off-h...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9344#issuecomment-152850346
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11410] [SQL] Add APIs to provide functi...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9364#issuecomment-152853348
  
**[Test build #44764 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44764/consoleFull)**
 for PR 9364 at commit 
[`98c05ae`](https://github.com/apache/spark/commit/98c05ae08281b14dd67151bf8edd55cb884d1061).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11401] [MLLIB] PMML export for Logistic...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9397#issuecomment-152841751
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10280][MLlib][PySpark][Docs] Add @since...

[yu-iskw]

Github user yu-iskw commented on the pull request:

https://github.com/apache/spark/pull/8690#issuecomment-152848582
  
Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Reduce numSlices for local metrics test of Spa...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9384#issuecomment-152848571
  
**[Test build #44760 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44760/consoleFull)**
 for PR 9384 at commit 
[`6c5c21c`](https://github.com/apache/spark/commit/6c5c21ca3f08ffa14c645b2778b08a15de009423).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-11435 Stop SparkContext at the end of su...

[tedyu]

Github user tedyu commented on the pull request:

https://github.com/apache/spark/pull/9384#issuecomment-152834222
  
The goal is to reduce memory consumption of local metrics test.
I want a QA run for latest change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11440][CORE][STREAMING][BUILD] Declare ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9396#issuecomment-152837808
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10280][MLlib][PySpark][Docs] Add @since...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8690#issuecomment-152848645
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10280][MLlib][PySpark][Docs] Add @since...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8690#issuecomment-152848657
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Reduce numSlices for local metrics test of Spa...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9384#issuecomment-152852553
  
**[Test build #44759 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44759/consoleFull)**
 for PR 9384 at commit 
[`7cc64f3`](https://github.com/apache/spark/commit/7cc64f3a5f86b371c6d2eb376e0fe4735306d9ea).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11236][CORE] Update Tachyon dependency ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9395#issuecomment-152828790
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11236][CORE] Update Tachyon dependency ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9395#issuecomment-152829433
  
**[Test build #44756 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44756/consoleFull)**
 for PR 9395 at commit 
[`6802ecd`](https://github.com/apache/spark/commit/6802ecd5c957a64bfcb11107ab8390a64a0d4ad6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11338][WebUI] Prepend app links on Hist...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9291#issuecomment-152829339
  
**[Test build #44754 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44754/consoleFull)**
 for PR 9291 at commit 
[`01d2f35`](https://github.com/apache/spark/commit/01d2f359956526990a18c2e7d093f9a2b4a07ad5).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10500][SPARKR][WIP] sparkr.zip cannot b...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9390#issuecomment-152829398
  
**[Test build #44753 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44753/consoleFull)**
 for PR 9390 at commit 
[`3fbb2db`](https://github.com/apache/spark/commit/3fbb2dbb3a2f7a1936b12cd5af7eab2f31d625e7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-11435 Stop SparkContext at the end of su...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9384#issuecomment-152834119
  
**[Test build #44757 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44757/consoleFull)**
 for PR 9384 at commit 
[`eb2d1b5`](https://github.com/apache/spark/commit/eb2d1b509fde34b106c10a01e91123e7fc6fd2e9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-11435 Stop SparkContext at the end of su...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/9384#issuecomment-152834147
  
The changes here are now unrelated it seems. Can you close this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10280][MLlib][PySpark][Docs] Add @since...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8690#issuecomment-152849701
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10280][MLlib][PySpark][Docs] Add @since...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8690#issuecomment-152849671
  
**[Test build #44762 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44762/consoleFull)**
 for PR 8690 at commit 
[`2c71cc2`](https://github.com/apache/spark/commit/2c71cc28e322e9dd011b97d76bff6c0d20fd0d31).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10280][MLlib][PySpark][Docs] Add @since...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8690#issuecomment-152849659
  
**[Test build #44763 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44763/consoleFull)**
 for PR 8690 at commit 
[`2c71cc2`](https://github.com/apache/spark/commit/2c71cc28e322e9dd011b97d76bff6c0d20fd0d31).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11389][WIP][CORE] Add support for off-h...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9344#issuecomment-152850349
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44755/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-11435 Stop SparkContext at the end of su...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9384#issuecomment-152833893
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-11435 Stop SparkContext at the end of su...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9384#issuecomment-152833927
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11338][WebUI] Prepend app links on Hist...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9291#issuecomment-152842124
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11338][WebUI] Prepend app links on Hist...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9291#issuecomment-152842091
  
**[Test build #44754 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44754/consoleFull)**
 for PR 9291 at commit 
[`01d2f35`](https://github.com/apache/spark/commit/01d2f359956526990a18c2e7d093f9a2b4a07ad5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11236][CORE] Update Tachyon dependency ...

[calvinjia]

Github user calvinjia commented on the pull request:

https://github.com/apache/spark/pull/9395#issuecomment-152847168
  
@yhuai 
Thanks for the the response! The last PR would pass with those parameters, 
but and would only fail if the entire dev/runtests cycle was ran (specifically 
if MIMA ran before). I've tested locally that the issues do not occur when 
running dev/runtests with this change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10280][MLlib][PySpark][Docs] Add @since...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8690#issuecomment-152848852
  
**[Test build #44762 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44762/consoleFull)**
 for PR 8690 at commit 
[`2c71cc2`](https://github.com/apache/spark/commit/2c71cc28e322e9dd011b97d76bff6c0d20fd0d31).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10280][MLlib][PySpark][Docs] Add @since...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8690#issuecomment-152848875
  
**[Test build #44763 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44763/consoleFull)**
 for PR 8690 at commit 
[`2c71cc2`](https://github.com/apache/spark/commit/2c71cc28e322e9dd011b97d76bff6c0d20fd0d31).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11112] DAG visualization: display RDD c...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9398#issuecomment-152848818
  
**[Test build #44761 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44761/consoleFull)**
 for PR 9398 at commit 
[`5b6c70e`](https://github.com/apache/spark/commit/5b6c70e804fe81ca127515f8e27cc05e99771a9b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11440][CORE][STREAMING][BUILD] Declare ...

[srowen]

GitHub user srowen opened a pull request:

https://github.com/apache/spark/pull/9396

[SPARK-11440][CORE][STREAMING][BUILD] Declare rest of @Experimental items 
non-experimental if they've existed since 1.2.0

Remove `@Experimental` annotations in core, streaming for items that 
existed in 1.2.0 or before. The changes are:

* SparkContext
  * binary{Files,Records} : 1.2.0
  * submitJob : 1.0.0
* JavaSparkContext
  * binary{Files,Records} : 1.2.0
* DoubleRDDFunctions, JavaDoubleRDD
  * {mean,sum}Approx : 1.0.0
* PairRDDFunctions, JavaPairRDD
  * sampleByKeyExact : 1.2.0
  * countByKeyApprox : 1.0.0
* PairRDDFunctions
  * countApproxDistinctByKey : 1.1.0
* RDD
  * countApprox, countByValueApprox, countApproxDistinct : 1.0.0
* JavaRDDLike
  * countApprox : 1.0.0
* PythonHadoopUtil.Converter : 1.1.0
* PortableDataStream : 1.2.0 (related to binaryFiles)
* BoundedDouble : 1.0.0
* PartialResult : 1.0.0
* StreamingContext, JavaStreamingContext
  * binaryRecordsStream : 1.2.0
* HiveContext
  * analyze : 1.2.0

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srowen/spark SPARK-11440

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9396.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9396


commit 2439d7105022aa67a635498fa2d5145a134150d7
Author: Sean Owen 
Date:   2015-11-01T15:35:50Z

Remove @Experimental annotations in core, streaming for items that existed 
in 1.2.0 or before




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11236][CORE] Update Tachyon dependency ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9395#issuecomment-152841958
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10500][SPARKR][WIP] sparkr.zip cannot b...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9390#issuecomment-152841957
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44753/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11236][CORE] Update Tachyon dependency ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9395#issuecomment-152841959
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44756/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Reduce numSlices for local metrics test of Spa...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9384#issuecomment-152852593
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44759/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11329] [SQL] Support star expansion for...

[yhuai]

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/9343#discussion_r43586963
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
 ---
@@ -166,26 +166,58 @@ abstract class Star extends LeafExpression with 
NamedExpression {
  * Represents all of the input attributes to a given relational operator, 
for example in
  * "SELECT * FROM ...".
  *
- * @param table an optional table that should be the target of the 
expansion.  If omitted all
- *  tables' columns are produced.
+ * This is also used to expand structs. For example:
+ * "SELECT record.* from (SELCCT struct(a,b,c) as record ...)
+ *
+ * @param target an optional name that should be the target of the 
expansion.  If omitted all
+ *  targets' columns are produced. This can either be a table 
name or struct name.
  */
-case class UnresolvedStar(table: Option[String]) extends Star with 
Unevaluable {
+case class UnresolvedStar(target: Option[String]) extends Star with 
Unevaluable {
+
+  override def expand(input: LogicalPlan, resolver: Resolver): 
Seq[NamedExpression] = {
+// First try to resolve the target as a struct expansion. That is try 
to see if it is
+// .*. If that fails, we'll try as a table expansion.
+// TODO: is this the order we want to resolve this?
--- End diff --

How about we try as a table expansion first since it is the current 
behavior? If there is a struct that has the same name with the table, we can 
use `name.name.*` to expand it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-6541 - Sort executors by ID (numeric)

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9165#issuecomment-152859815
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9218#issuecomment-152861491
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9218#issuecomment-152861497
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11338][WebUI] Prepend app links on Hist...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/9291


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10978] [SQL] Allow data sources to elim...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9399#issuecomment-152865071
  
**[Test build #44768 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44768/consoleFull)**
 for PR 9399 at commit 
[`16f3ca3`](https://github.com/apache/spark/commit/16f3ca3c34533922c2d6d4dbedf9818dd281053f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-11382 Replace example code in mllib-deci...

[yinxusen]

Github user yinxusen commented on the pull request:

https://github.com/apache/spark/pull/9378#issuecomment-152866685
  
@gliptak Why the code changes do not match with the title? I think the 
SPARK-11382 should fix mllib-decision-tree.md and mllib-ensembles.md.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11389][WIP][CORE] Add support for off-h...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9344#issuecomment-152867810
  
**[Test build #44770 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44770/consoleFull)**
 for PR 9344 at commit 
[`144e680`](https://github.com/apache/spark/commit/144e68042fb773295d69f57bd785cc1eeed4659c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-11442 Reduce numSlices for local metrics...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9384#issuecomment-152868378
  
**[Test build #44772 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44772/consoleFull)**
 for PR 9384 at commit 
[`1318676`](https://github.com/apache/spark/commit/1318676750234d9d7ec18acc7a5fe241062b62ce).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-11442 Reduce numSlices for local metrics...

[tedyu]

Github user tedyu commented on the pull request:

https://github.com/apache/spark/pull/9384#issuecomment-152868318
  
bq. what does this have to do with memory
The answer is in SaveStageAndTaskInfo. See the following
```
   * A simple listener that saves all task infos and task metrics.
   */
  private class SaveStageAndTaskInfo extends SparkListener {
val stageInfos = mutable.Map[StageInfo, Seq[(TaskInfo, TaskMetrics)]]()
var taskInfoMetrics = mutable.Buffer[(TaskInfo, TaskMetrics)]()
```
bq. Why is it valid to reduce the number of slices in this test?

Let me dig into the first checkin of this test to see why 64 slices were 
used.
As recent test runs showed, the invariant of the test is still true for 16 
slices.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9218#issuecomment-152868688
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11443] Reserve space lines

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9400#issuecomment-152868699
  
**[Test build #44773 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44773/consoleFull)**
 for PR 9400 at commit 
[`d35d9c1`](https://github.com/apache/spark/commit/d35d9c1d0850a616ead8b0015526a16d539faf86).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK][SPARK-10842]Eliminate creating duplica...

[markhamstra]

Github user markhamstra commented on the pull request:

https://github.com/apache/spark/pull/8923#issuecomment-152869987
  
I prefer to fix this by changing the result type of 
`getAncestorShuffleDependencies(rdd: RDD[_])` to be `Set[ShuffleDependency[_, 
_, _]]` and the result of that method from `parents` to `parents.toSet`.  It 
was never the intent, nor is it useful, for `getAncestorShuffleDependencies` to 
produce duplicates.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11314] [YARN] add service API and test ...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/9182#issuecomment-152870622
  
BTW, if you're keeping `bindToYarn` and friends, you could change 
`applicationId` and `applicationAttemptId` to return the values you're setting 
there, which also means you could have a single implementation for the two 
methods (instead of the current separate impls for client and cluster modes).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11407][SPARKR] Add doc for running from...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9401#issuecomment-152870725
  
**[Test build #44774 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44774/consoleFull)**
 for PR 9401 at commit 
[`a048574`](https://github.com/apache/spark/commit/a0485741f86656c0f4c5a588fd69598f04f49cd1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11424] Guard against double-close() of ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9388#issuecomment-152871725
  
**[Test build #44765 timed 
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44765/consoleFull)**
 for PR 9388 at commit 
[`11d5e70`](https://github.com/apache/spark/commit/11d5e70e2547f9d7ad431b60ed5c3cfbb272c8c8)
 after a configured wait of `175m`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10978] [SQL] Allow data sources to elim...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9399#issuecomment-152872738
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10978] [SQL] Allow data sources to elim...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9399#issuecomment-152872862
  
**[Test build #44776 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44776/consoleFull)**
 for PR 9399 at commit 
[`fec7d25`](https://github.com/apache/spark/commit/fec7d257f7ba5c8d797d6ed28ce2c93adabb7533).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9722][ML] Pass random seed to spark.ml ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9402#issuecomment-152874464
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9858][SPARK-9859][SPARK-9861][SQL] Add ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9276#issuecomment-152877559
  
**[Test build #44781 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44781/consoleFull)**
 for PR 9276 at commit 
[`890521e`](https://github.com/apache/spark/commit/890521e54d81f8ef8172a43fff50f824a7fec1be).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10978] [SQL] Allow data sources to elim...

[yhuai]

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/9399#discussion_r43590253
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
 ---
@@ -266,26 +267,39 @@ private[sql] object DataSourceStrategy extends 
Strategy with Logging {
   relation,
   projects,
   filterPredicates,
-  (requestedColumns, pushedFilters) => {
-scanBuilder(requestedColumns, selectFilters(pushedFilters).toArray)
+  (requestedColumns, _, pushedFilters) => {
+scanBuilder(requestedColumns, pushedFilters.toArray)
   })
   }
 
   // Based on Catalyst expressions.
   protected def pruneFilterProjectRaw(
-  relation: LogicalRelation,
-  projects: Seq[NamedExpression],
-  filterPredicates: Seq[Expression],
-  scanBuilder: (Seq[Attribute], Seq[Expression]) => RDD[InternalRow]) 
= {
+relation: LogicalRelation,
+projects: Seq[NamedExpression],
+filterPredicates: Seq[Expression],
+scanBuilder: (Seq[Attribute], Seq[Expression], Seq[Filter]) => 
RDD[InternalRow]) = {
--- End diff --

It is not obvious that we need both `Seq[Expression]` and `Seq[Filter]`. 
Can you add comments to explain what are these? Also, I feel filters in the 
catalyst form `Seq[Expression]` should be equivalent with filters in the public 
`Filter` form.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10978] [SQL] Allow data sources to elim...

[yhuai]

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/9399#discussion_r43590460
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
 ---
@@ -295,18 +309,20 @@ private[sql] object DataSourceStrategy extends 
Strategy with Logging {
   val requestedColumns =
 projects.asInstanceOf[Seq[Attribute]] // Safe due to if above.
   .map(relation.attributeMap)// Match original case of 
attributes.
+  .filterNot(handledSet.contains)
 
   val scan = execution.PhysicalRDD.createFromDataSource(
 projects.map(_.toAttribute),
-scanBuilder(requestedColumns, pushedFilters),
+scanBuilder(requestedColumns, candidatePredicates, pushedFilters),
--- End diff --

I think I understand what's going on. Actually, `pushedFilters` also 
contains those filters that cannot be handled by a data source and 
`candidatePredicates` contains filters that cannot be converted to public 
Filter interface.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10978] [SQL] Allow data sources to elim...

[yhuai]

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/9399#discussion_r43590582
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/sources/FilteredScanSuite.scala ---
@@ -44,16 +44,46 @@ case class SimpleFilteredScan(from: Int, to: 
Int)(@transient val sqlContext: SQL
   StructField("b", IntegerType, nullable = false) ::
   StructField("c", StringType, nullable = false) :: Nil)
 
+  /**
+   * Given an array of [[Filter]]s, returns an array of [[Filter]]s that 
this data source relation
+   * cannot handle.  Spark SQL will apply all returned [[Filter]]s against 
rows returned by this
+   * data source relation.
+   *
+   * @since 1.6.0
+   */
+  override def unhandledFilters(filters: Array[Filter]): Array[Filter] = {
+def unhandled(filter: Filter): Boolean = {
+  filter match {
+case EqualTo("b", v) => true
+case EqualNullSafe("b", v) => true
+case LessThan("b", v: Int) => true
+case LessThanOrEqual("b", v: Int) => true
+case GreaterThan("b", v: Int) => true
+case GreaterThanOrEqual("b", v: Int) => true
+case In("b", values) => true
+case IsNull("b") => true
+case IsNotNull("b") => true
+case Not(pred) => unhandled(pred)
+case And(left, right) => unhandled(left) || unhandled(right)
+case Or(left, right) => unhandled(left) || unhandled(right)
+case _ => false
--- End diff --

Which tests trigger this case?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10978] [SQL] Allow data sources to elim...

[yhuai]

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/9399#discussion_r43590585
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/sources/ParquetHadoopFsRelationSuite.scala
 ---
@@ -145,14 +145,16 @@ class ParquetHadoopFsRelationSuite extends 
HadoopFsRelationTest {
 
   test("SPARK-10334 Projections and filters should be kept in physical 
plan") {
 withTempPath { dir =>
-  val path = dir.getCanonicalPath
+  withSQLConf(SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "false") {
--- End diff --

why do we change this setting?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9722][ML] Pass random seed to spark.ml ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9402#issuecomment-152879812
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44777/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10978] [SQL] Allow data sources to elim...

[yhuai]

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/9399#discussion_r43590547
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/sources/FilteredScanSuite.scala ---
@@ -101,6 +130,10 @@ object FiltersPushed {
   var list: Seq[Filter] = Nil
 }
 
+object ColumnsRequired {
+  var set: Set[String] = Set.empty
+}
--- End diff --

comment?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org