[GitHub] spark pull request: [SPARK-2298] Encode stage attempt in SparkList...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1545#issuecomment-52740586 **Tests timed out** after a configured wait of `120m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3146][Streaming] Improve the flexibilit...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2053#issuecomment-52740578 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18928/consoleFull) for PR 2053 at commit [`4c3be07`](https://github.com/apache/spark/commit/4c3be07474ab4b7b6e99b38a87eb32b6d81a7b2a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3146][Streaming] Improve the flexibilit...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2053#issuecomment-52740591 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18928/consoleFull) for PR 2053 at commit [`4c3be07`](https://github.com/apache/spark/commit/4c3be07474ab4b7b6e99b38a87eb32b6d81a7b2a). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2849] Handle driver configs separately ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52740893 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18929/consoleFull) for PR 1845 at commit [`9a778f6`](https://github.com/apache/spark/commit/9a778f66d812886b3322b32ae92fccf9d8415657). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3054][STREAMING] Add unit tests for Spa...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1958#issuecomment-52740904 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3146][Streaming] Improve the flexibilit...
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/2053#issuecomment-52741181 Seems some temporary files messed up the test environment... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3054][STREAMING] Add unit tests for Spa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1958#issuecomment-52741251 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18930/consoleFull) for PR 1958 at commit [`e3110b9`](https://github.com/apache/spark/commit/e3110b9b551fb0019d6095a16504e8b25d70f962). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add some shuffle configurations in doc
Github user CrazyJvm commented on the pull request: https://github.com/apache/spark/pull/2031#issuecomment-52741370 @colorant thanks, I've not noticed `toLowerCase ` before : ) . already modified --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/2030#discussion_r16461557
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -109,99 +155,30 @@ private[spark] class TorrentBroadcast[T: ClassTag](
private def readObject(in: ObjectInputStream) {
in.defaultReadObject()
TorrentBroadcast.synchronized {
--- End diff --
Isn't this logic almost exactly what the CacheManager does already, fine
grained locking on reconstructing specific blocks?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add some shuffle configurations in doc
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2031#issuecomment-52741646 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18932/consoleFull) for PR 2031 at commit [`181c512`](https://github.com/apache/spark/commit/181c512c7d2340d0e7252793622cb1faddf30e78). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add some shuffle configurations in doc
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2031#issuecomment-52741632 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18932/consoleFull) for PR 2031 at commit [`181c512`](https://github.com/apache/spark/commit/181c512c7d2340d0e7252793622cb1faddf30e78). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2298] Encode stage attempt in SparkList...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1545#issuecomment-52741946 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2030#discussion_r16461721
--- Diff:
core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala ---
@@ -109,99 +155,30 @@ private[spark] class TorrentBroadcast[T: ClassTag](
private def readObject(in: ObjectInputStream) {
in.defaultReadObject()
TorrentBroadcast.synchronized {
--- End diff --
maybe we can generalize that to provide a lock manager, similar to what
dbms would do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/2030#issuecomment-52742011 @JoshRosen Did you ever see a significant difference between pre-PR and post-PR numbers? 1.0.2 was always much worse in all your graphs, but was this PR significant for larger blocks? By the way, 24 hour turnarounds on critical PRs that are going in at the end of a release cycle seem like a really bad way to preserve stability. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2030#issuecomment-52742166 FYI we actually reran all the mllib and scheduler throughput tests for this PR to verify. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2298] Encode stage attempt in SparkList...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1545#issuecomment-52742343 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18933/consoleFull) for PR 1545 at commit [`40a6bd5`](https://github.com/apache/spark/commit/40a6bd55f32273bab93fac504f26473a9c0364f3). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2298] Encode stage attempt in SparkList...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1545#issuecomment-52742350 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18933/consoleFull) for PR 1545 at commit [`40a6bd5`](https://github.com/apache/spark/commit/40a6bd55f32273bab93fac504f26473a9c0364f3). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SparkListenerTaskStart(stageId: Int, stageAttemptId: Int, taskInfo: TaskInfo)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2298] Encode stage attempt in SparkList...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1545#issuecomment-52742473 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2849] Handle driver configs separately ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52742587 Hi everyone, thanks for your comments. I have tested the latest commit and I think this is ready to go in my opinion. I have fixed the PySpark behavior where the subprocess doesn't actually terminate in a similar way Py4j's `JavaGateway` handles it (see `--die-on-broken-pipe`, which we currently use for the direct `SparkSubmit` code path). I have verified that the precedence order of various driver configs are as expected. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3124] [SQL] Fix the assembly jar confli...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2035#issuecomment-52742669 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18935/consoleFull) for PR 2035 at commit [`182b52b`](https://github.com/apache/spark/commit/182b52bc59231395a336f41c7d8bfded1a9ca5b0). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2312] Logging Unhandled messages
Github user isaias commented on the pull request: https://github.com/apache/spark/pull/2048#issuecomment-52742677 @rxin, the trait ActorLogReceive gets the concrete implementation through receiveWithLogging function overriding of implementing Actor and applying it, so I think that on trait I can't identify the unhandled message of concrete implementation. Maybe I'm missing something, but I'll review. BR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2849] Handle driver configs separately ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52742757 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18934/consoleFull) for PR 1845 at commit [`24dba60`](https://github.com/apache/spark/commit/24dba6067b33719e7eb25b0fb536bc2df72d34a4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2298] Encode stage attempt in SparkList...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1545#issuecomment-52742759 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18936/consoleFull) for PR 1545 at commit [`40a6bd5`](https://github.com/apache/spark/commit/40a6bd55f32273bab93fac504f26473a9c0364f3). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SparkListenerTaskStart(stageId: Int, stageAttemptId: Int, taskInfo: TaskInfo)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2298] Encode stage attempt in SparkList...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1545#issuecomment-52742752 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18936/consoleFull) for PR 1545 at commit [`40a6bd5`](https://github.com/apache/spark/commit/40a6bd55f32273bab93fac504f26473a9c0364f3). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2849] Handle driver configs separately ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52742770 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18934/consoleFull) for PR 1845 at commit [`24dba60`](https://github.com/apache/spark/commit/24dba6067b33719e7eb25b0fb536bc2df72d34a4). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2312] Logging Unhandled messages
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/2048#issuecomment-52742926
You just need to change
```scala
override def isDefinedAt(o: Any): Boolean =
_receiveWithLogging.isDefinedAt(o)
```
to
```scala
override def isDefinedAt(o: Any): Boolean = {
val handled = _receiveWithLogging.isDefinedAt(o)
if (!handled) {
log ...
}
handled
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2849] Handle driver configs separately ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52743041 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2849] Handle driver configs separately ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52743537 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18937/consoleFull) for PR 1845 at commit [`24dba60`](https://github.com/apache/spark/commit/24dba6067b33719e7eb25b0fb536bc2df72d34a4). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2849] Handle driver configs separately ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52743530 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18937/consoleFull) for PR 1845 at commit [`24dba60`](https://github.com/apache/spark/commit/24dba6067b33719e7eb25b0fb536bc2df72d34a4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2849] Handle driver configs separately ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52743675 Hm, test this please! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2849] Handle driver configs separately ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52743953 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18938/consoleFull) for PR 1845 at commit [`24dba60`](https://github.com/apache/spark/commit/24dba6067b33719e7eb25b0fb536bc2df72d34a4). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2849] Handle driver configs separately ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52743943 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18938/consoleFull) for PR 1845 at commit [`24dba60`](https://github.com/apache/spark/commit/24dba6067b33719e7eb25b0fb536bc2df72d34a4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2849] Handle driver configs separately ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52744147 Jenkins is failing because of some mllib / streaming leftover state --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2849] Handle driver configs separately ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52744175 Let me try this once more. Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2849] Handle driver configs separately ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52744341 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18939/consoleFull) for PR 1845 at commit [`24dba60`](https://github.com/apache/spark/commit/24dba6067b33719e7eb25b0fb536bc2df72d34a4). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2312] Logging Unhandled messages
Github user isaias commented on the pull request: https://github.com/apache/spark/pull/2048#issuecomment-52744357 @rxin Very nice, it works. Should be better to close this PR and send another just with the trait changes? BR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2312] Logging Unhandled messages
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2048#issuecomment-52745114 Sure. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2312] Logging Unhandled messages
Github user isaias closed the pull request at: https://github.com/apache/spark/pull/2048 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2312] Logging Unhandled messages
Github user isaias commented on the pull request: https://github.com/apache/spark/pull/2048#issuecomment-52745159 Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2849] Handle driver configs separately ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52745225 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18929/consoleFull) for PR 1845 at commit [`9a778f6`](https://github.com/apache/spark/commit/9a778f66d812886b3322b32ae92fccf9d8415657). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3135] Created ByteArrayChunkOutputStrea...
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/2054 [SPARK-3135] Created ByteArrayChunkOutputStream and used it to avoid mem copy in TorrentBroadcast. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark ByteArrayChunkOutputStream Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2054.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2054 commit 8f1a8eb59af3760651a391ea0fc76255d855cf2e Author: Reynold Xin Date: 2014-08-20T07:59:01Z [SPARK-3135] Created ByteArrayChunkOutputStream and used it to avoid memory copy in TorrentBroadcast. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3054][STREAMING] Add unit tests for Spa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1958#issuecomment-52745587 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18930/consoleFull) for PR 1958 at commit [`e3110b9`](https://github.com/apache/spark/commit/e3110b9b551fb0019d6095a16504e8b25d70f962). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1853] Show Streaming application code c...
Github user mubarak commented on a diff in the pull request:
https://github.com/apache/spark/pull/1723#discussion_r16463252
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala ---
@@ -106,6 +106,27 @@ abstract class DStream[T: ClassTag] (
/** Return the StreamingContext associated with this DStream */
def context = ssc
+ /* Find the creation callSite */
+ val creationSite = Utils.getCallSite
+
+ /* Store the RDD creation callSite in threadlocal */
+ private def setRDDCreationCallSite() = {
+ssc.sparkContext.setLocalProperty(Utils.CALL_SITE_SHORT,
creationSite.shortForm)
--- End diff --
Updated
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3135] Created ByteArrayChunkOutputStrea...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2054#issuecomment-52746053 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18942/consoleFull) for PR 2054 at commit [`ee88e73`](https://github.com/apache/spark/commit/ee88e7338c9d07da9a3e647eb4ba36302c29e3f6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1853] Show Streaming application code c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1723#issuecomment-52746058 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18943/consoleFull) for PR 1723 at commit [`491a1eb`](https://github.com/apache/spark/commit/491a1eb23aa94919a9a06a84a9cf3cab118fa628). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1853] Show Streaming application code c...
Github user mubarak commented on the pull request: https://github.com/apache/spark/pull/1723#issuecomment-52746109 Jenkins, this is ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3135] Avoid extra mem copy in TorrentBr...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2054#discussion_r16463419
--- Diff:
core/src/main/scala/org/apache/spark/util/io/ByteArrayChunkOutputStream.scala
---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.io
+
+import java.io.OutputStream
+
+import scala.collection.mutable.ArrayBuffer
+
+
+/**
+ * An OutputStream that writes to fixed-size chunks of byte arrays.
+ */
+private[spark]
+class ByteArrayChunkOutputStream(chunkSize: Int) extends OutputStream {
+
+ private val chunks = new ArrayBuffer[Array[Byte]]
+
+ private var lastChunkIndex = -1
+ private var position = chunkSize
+
+ @inline
+ private def allocateNewChunkIfNeeded(): Unit = {
+if (position == chunkSize) {
+ chunks += new Array[Byte](chunkSize)
+ lastChunkIndex += 1
+ position = 0
+}
+ }
+
+ def toArrays: Array[Array[Byte]] = {
+if (lastChunkIndex == -1) {
+ new Array[Array[Byte]](0)
+} else {
+ val ret = new Array[Array[Byte]](chunks.size)
+ for (i <- 0 until chunks.size - 1) {
+ret(i) = chunks(i)
+ }
+ if (position == chunkSize) {
+ret(lastChunkIndex) = chunks(lastChunkIndex)
+ } else {
+ret(lastChunkIndex) = new Array[Byte](position)
+System.arraycopy(chunks(lastChunkIndex), 0, ret(lastChunkIndex),
0, position)
--- End diff --
note to self: add comment explaining why we don't return bytebuffers with
limit set
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3146][Streaming] Improve the flexibilit...
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/2053#issuecomment-52746388 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3146][Streaming] Improve the flexibilit...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2053#issuecomment-52746860 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18944/consoleFull) for PR 2053 at commit [`4c3be07`](https://github.com/apache/spark/commit/4c3be07474ab4b7b6e99b38a87eb32b6d81a7b2a). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3146][Streaming] Improve the flexibilit...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2053#issuecomment-52746851 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18944/consoleFull) for PR 2053 at commit [`4c3be07`](https://github.com/apache/spark/commit/4c3be07474ab4b7b6e99b38a87eb32b6d81a7b2a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3124] [SQL] Fix the assembly jar confli...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2035#issuecomment-52747253 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18935/consoleFull) for PR 2035 at commit [`182b52b`](https://github.com/apache/spark/commit/182b52bc59231395a336f41c7d8bfded1a9ca5b0). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3138][SQL] sqlContext.parquetFile shoul...
Github user chutium commented on the pull request: https://github.com/apache/spark/pull/2044#issuecomment-52747630 actually i totally agree with you @liancheng , and it is not "try to handle", it is more like "try to ignore useless check" :) it is handled by children.find(...) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3124] [SQL] Fix the assembly jar confli...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/2035#issuecomment-52747698 test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2312] Logging Unhandled messages
GitHub user isaias opened a pull request: https://github.com/apache/spark/pull/2055 [SPARK-2312] Logging Unhandled messages The previous solution has changed based on https://github.com/apache/spark/pull/2048 discussions. You can merge this pull request into a Git repository by running: $ git pull https://github.com/isaias/spark SPARK-2312 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2055.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2055 commit f34177754550f27445ac9ac7313fe1b6b4ee7e21 Author: Isaias Barroso Date: 2014-08-20T08:28:33Z [SPARK-2312] Logging Unhandled messages --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3124] [SQL] Fix the assembly jar confli...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2035#issuecomment-52748199 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18946/consoleFull) for PR 2035 at commit [`81fc0d7`](https://github.com/apache/spark/commit/81fc0d7c04bcd7cb09c2c7c208fa76bc5f8816a0). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3146][Streaming] Improve the flexibilit...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2053#issuecomment-52748195 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18945/consoleFull) for PR 2053 at commit [`6f9baad`](https://github.com/apache/spark/commit/6f9baade812d375cff072378918a4d0ccdf906af). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3124] [SQL] Fix the assembly jar confli...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2035#issuecomment-52748211 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18946/consoleFull) for PR 2035 at commit [`81fc0d7`](https://github.com/apache/spark/commit/81fc0d7c04bcd7cb09c2c7c208fa76bc5f8816a0). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3146][Streaming] Improve the flexibilit...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2053#issuecomment-52748206 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18945/consoleFull) for PR 2053 at commit [`6f9baad`](https://github.com/apache/spark/commit/6f9baade812d375cff072378918a4d0ccdf906af). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2312] Logging Unhandled messages
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2055#issuecomment-52748507 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3015] Block on cleaning tasks to preven...
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/1931#discussion_r16464399
--- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala ---
@@ -66,10 +66,15 @@ private[spark] class ContextCleaner(sc: SparkContext)
extends Logging {
/**
* Whether the cleaning thread will block on cleanup tasks.
- * This is set to true only for tests.
+ *
+ * Due to SPARK-3015, this is set to true by default. This is intended
to be only a temporary
+ * workaround for the issue, which is ultimately caused by the way the
BlockManager actors
+ * issue inter-dependent blocking Akka messages to each other at high
frequencies. This happens,
+ * for instance, when the driver performs a GC and cleans up all
broadcast blocks that are no
+ * longer in scope.
*/
private val blockOnCleanupTasks = sc.conf.getBoolean(
-"spark.cleaner.referenceTracking.blocking", false)
--- End diff --
The changes will not solve the problem here. see.
[BlockManagerMasterActor.scala#L165](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala#L165)
```scala
private def removeShuffle(shuffleId: Int): Future[Seq[Boolean]] = {
// Nothing to do in the BlockManagerMasterActor data structures
import context.dispatcher
val removeMsg = RemoveShuffle(shuffleId)
Future.sequence(
blockManagerInfo.values.map { bm =>
// Here has set the akkaTimeout
bm.slaveActor.ask(removeMsg)(akkaTimeout).mapTo[Boolean]
}.toSeq
)
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3138][SQL] sqlContext.parquetFile shoul...
Github user chutium commented on the pull request: https://github.com/apache/spark/pull/2044#issuecomment-52748630 actually i totally agree with you @liancheng , this was reported in user list http://apache-spark-user-list.1001560.n3.nabble.com/sqlContext-parquetFile-path-fails-if-path-is-a-file-but-succeeds-if-a-directory-td12345.html i think he is also right, parameter of sqlContext.parquetFile should be a dir of parquet files (at better with a _metadata file in it), but this should not be a mandatory requirement, our javadoc https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/SQLContext.html#parquetFile(java.lang.String) says also, this method "Loads a Parquet file", not "loads a dir of parqauet files"... otherwise maybe we should rename this method to sqlContext.parquetDir or somthing and this PR is not "try to handle", it is more like "try to ignore useless check" :) it is already handled by children.find(...) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3126][SPARK-3127][SQL] Fixed HiveThrift...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/2036#issuecomment-52748646 The unit test failure probably due to the assembly jar with duplicated netty jars (netty.3.2.2.Final.jar & netty.3.6.6.Final.jar, the class `NioWorker` with different method signatures), I've created another PR https://github.com/apache/spark/pull/2035 to fix this. And of course the `"` should be removed as @liancheng did already. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3146][Streaming] Improve the flexibilit...
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/2053#issuecomment-52749189 Hey @tdas , mind taking a look at this? Looks like checkpoint file messed up the environment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1853] Show Streaming application code c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1723#issuecomment-52750288 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18943/consoleFull) for PR 1723 at commit [`491a1eb`](https://github.com/apache/spark/commit/491a1eb23aa94919a9a06a84a9cf3cab118fa628). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3139] Akka timeouts from ContextCleaner...
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/2056 [SPARK-3139] Akka timeouts from ContextCleaner when cleaning shuffles You can merge this pull request into a Git repository by running: $ git pull https://github.com/witgo/spark SPARK-3139 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2056.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2056 commit a49bc80deac4a4ca1c288e6ed99324f7fb92bc46 Author: GuoQiang Li Date: 2014-08-20T08:58:13Z Akka timeouts from ContextCleaner when cleaning shuffles --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3135] Avoid extra mem copy in TorrentBr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2054#issuecomment-52751016 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18942/consoleFull) for PR 2054 at commit [`ee88e73`](https://github.com/apache/spark/commit/ee88e7338c9d07da9a3e647eb4ba36302c29e3f6). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class ByteArrayChunkOutputStream(chunkSize: Int) extends OutputStream ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3139] Akka timeouts from ContextCl...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2056#issuecomment-52751046 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18947/consoleFull) for PR 2056 at commit [`a49bc80`](https://github.com/apache/spark/commit/a49bc80deac4a4ca1c288e6ed99324f7fb92bc46). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3139] Akka timeouts from ContextCl...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2056#issuecomment-52751321 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18947/consoleFull) for PR 2056 at commit [`a49bc80`](https://github.com/apache/spark/commit/a49bc80deac4a4ca1c288e6ed99324f7fb92bc46). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2312] Logging Unhandled messages
Github user isaias commented on the pull request: https://github.com/apache/spark/pull/2055#issuecomment-52752384 After send the PR I realized that I could have written some tests based on underlyingActor and isDefinedAt, if is valuable I can do it. BR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3139] Akka timeouts from ContextCl...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2056#issuecomment-52752829 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18948/consoleFull) for PR 2056 at commit [`f81c6a9`](https://github.com/apache/spark/commit/f81c6a913738098ae6a72cdd78ea2e01f5c27e9a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3139] Akka timeouts from ContextCl...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2056#issuecomment-52752840 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18948/consoleFull) for PR 2056 at commit [`f81c6a9`](https://github.com/apache/spark/commit/f81c6a913738098ae6a72cdd78ea2e01f5c27e9a). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3139] Akka timeouts from ContextCl...
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/2056#issuecomment-52753087 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3139] Akka timeouts from ContextCl...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2056#issuecomment-52753280 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18949/consoleFull) for PR 2056 at commit [`f81c6a9`](https://github.com/apache/spark/commit/f81c6a913738098ae6a72cdd78ea2e01f5c27e9a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK_LOGFILE and SPARK_ROOT_LOGGER no longer ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2057#issuecomment-52755341 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK_LOGFILE and SPARK_ROOT_LOGGER no longer ...
GitHub user scwf opened a pull request: https://github.com/apache/spark/pull/2057 SPARK_LOGFILE and SPARK_ROOT_LOGGER no longer need in spark-daemon.sh You can merge this pull request into a Git repository by running: $ git pull https://github.com/scwf/spark patch-7 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2057.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2057 commit 1b7b9a56d60719ed0e711c38827d3c806c1c4cb0 Author: wangfei Date: 2014-08-20T09:46:55Z SPARK_LOGFILE and SPARK_ROOT_LOGGER no longer need in spark-daemon.sh --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3012] Standardized Distance Functions b...
Github user yu-iskw commented on the pull request: https://github.com/apache/spark/pull/1964#issuecomment-52756284 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3012] Standardized Distance Functions b...
Github user yu-iskw commented on the pull request: https://github.com/apache/spark/pull/1964#issuecomment-52756243 Hi @erikerlandson, Thank you for your comment. I modify the code according to your advice. Would you please check it. - DistanceMeasure trait extends Function2[Vector, Vector, Double] - Add implicit method for DistanceMeasure - Add DistanceMetric trait for not distance function -- ex) CosineDistanceMeasure, TanimotoDistanceMeasure, SquaredEuclideanMeasure are subclass it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [HOTFIX][Streaming][MLlib] use temp folder for...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2046#discussion_r16468398
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala ---
@@ -119,7 +119,12 @@ trait TestSuiteBase extends FunSuite with
BeforeAndAfter with Logging {
def batchDuration = Seconds(1)
// Directory where the checkpoint data will be saved
- def checkpointDir = "checkpoint"
+ lazy val checkpointDir = {
+val dir = Files.createTempDir()
--- End diff --
Ideally, delete this explicitly and recursively in the 'after' function. It
could fail to be deleted if the JVM crashes later on. Minor though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL]Add "Dynamic Partition" suppo...
Github user baishuo commented on the pull request:
https://github.com/apache/spark/pull/1919#issuecomment-52758525
Here I try to explain my design idea(the code is mostly in
InsertIntoHiveTable.scala) :
lets assume there is a table called table1,which has 2 columns:col1,col2,
and two partitions: part1, part2.
ONE:
In case of just insert data to a static partition,I find when
"saveAsHiveFile" finished, the data was wroten to a temporary location, then
directory like: /tmp/hive-root/hive_/-ext-1,lets call it TMPLOCATION.
And under TMPLOCATION, there is sub directory /part1=.../part2=... , all data
was store under TMPLOCATION/part1=.../part2=... , then spark will call hive api
"loadPartition" to move the files to
{hivewarehouse}/{tablename}/part1=.../part2=... and update the metadata. then
the whole progress is OK.
If we what to implement the "dynamic partiton function", we need to use
hive api "loadDynamicPartitions" to move data and update metadata. But the
requirement of directory formate for "loadDynamicPartitions" is a little
difference to "loadPartition":
1: In case of one static partition and one dynamic partition (HQL like "
insert overwrite table table1 partition(part1=val1,part2) select a,b,c from
..."), loadDynamicPartitions need the tmp data located at TMPLOCATION/part2=c1,
TMPLOCATION/part2=c2 ..., And loadDynamicPartitions will move them to
{hivewarehouse}/{tablename}/part1=val1/part2=c1,
{hivewarehouse}/{tablename}/part1=val1/part2=c2 , and update the metadata.
Note that in this case loadDynamicPartitions do note need the subdir like
part1=val1 under TMPLOCATION
2: In case of zero static partition and 2 dynamic partition (HQL like "
insert overwrite table table1 partition(part1,part2) select a,b,x,c from
..."), loadDynamicPartitions need the tmp data located at
TMPLOCATION/part1=../part2=c1, TMPLOCATION/part1=../part2=c2 ..., And
loadDynamicPartitions will move them to
{hivewarehouse}/{tablename}/part1=../part2=...,
So if there have static partition in HQL determine how we create subdir
under TMPLOCATION. That why the function "getDynamicPartDir" exist.
TWO:
where shall we call the "getDynamicPartDir"? must a location that we can
get the values for dynamic partiton. so we call this function at "iter.map {
row =>..." in the closure of "val rdd = childRdd.mapPartitions". when we get
the row, we can get the values for dynamic partiton. after we get the
dynamicPartPath by function getDynamicPartDir, we can pass it to next RDD by
the output this RDD: serializer.serialize(outputData, standardOI) ->
dynamicPartPath. (for the static partiton,dynamicPartPath is null)
when the next rdd (closure in writeToFile) get the data and
dynamicPartPath, we can check if the dynamicPartPath equals null. if not null.
we check if there is already a corresponding writer exist in writerMap which
store all writer for each partition. if there is. we use this writer to write
the record. that ensure the data belongs to same partition will be wroten to
the same directory.
loadDynamicPartitions require there is no other files under TMPLOCATION
except the subdir for dynamic partition. that why there are several "if
(dynamicPartNum == 0)" in writeToFile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3139] Akka timeouts from ContextCl...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2056#issuecomment-52759438 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18949/consoleFull) for PR 2056 at commit [`f81c6a9`](https://github.com/apache/spark/commit/f81c6a913738098ae6a72cdd78ea2e01f5c27e9a). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3054][STREAMING] Add unit tests for Spa...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1958#issuecomment-52761646 Alright, I am merging this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3054][STREAMING] Add unit tests for Spa...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1958 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3069 [DOCS] Build instructions in README...
Github user bbossy commented on the pull request: https://github.com/apache/spark/pull/2014#issuecomment-52769500 Yes, LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2987] - Adjust build system and tests t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1905#issuecomment-52769948 QA tests have started for PR 1905. This patch DID NOT merge cleanly! View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18950/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2987] - Adjust build system and tests t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1905#issuecomment-52772733 QA tests have started for PR 1905. This patch DID NOT merge cleanly! View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18951/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update global variables of HttpBroadcast so th...
GitHub user YanTangZhai opened a pull request: https://github.com/apache/spark/pull/2058 Update global variables of HttpBroadcast so that multiple SparkContexts can coexist Update global variables of HttpBroadcast so that multiple SparkContexts can coexist You can merge this pull request into a Git repository by running: $ git pull https://github.com/YanTangZhai/spark httpbroadcast Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2058.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2058 commit cdef539abc5d2d42d4661373939bdd52ca8ee8e6 Author: YanTangZhai Date: 2014-08-06T13:07:08Z Merge pull request #1 from apache/master update commit b9921f37aa13620c5bea82512b63e2b0a73b5ffa Author: yantangzhai Date: 2014-08-20T12:56:00Z Update global variables of HttpBroadcast so that multiple SparkContexts can coexist commit 07d719ff7d77a66a4b67ef84ba9d4e5e881391fb Author: yantangzhai Date: 2014-08-20T12:57:19Z Update global variables of HttpBroadcast so that multiple SparkContexts can coexist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update global variables of HttpBroadcast so th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2058#issuecomment-52775792 QA results for PR 2058:- This patch FAILED unit tests.For more information see test ouptut:https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18952/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update global variables of HttpBroadcast so th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2058#issuecomment-52775780 QA tests have started for PR 2058. This patch DID NOT merge cleanly! View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18952/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2960] Support executing Spark from syml...
Github user roji commented on the pull request: https://github.com/apache/spark/pull/1875#issuecomment-52777250 @mateiz, added a commit for the sbin scripts. In general the scripts could use a bit of cleanup - the bin and sbin scripts work a bit differently, the sbin-specific spark-config.sh doesn't do much anymore, SPARK_HOME and SPARK_PREFIX are used interchangeably... But it isn't very important. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2960] Support executing Spark from syml...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1875#issuecomment-52777692 QA tests have started for PR 1875. This patch DID NOT merge cleanly! View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18953/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2960] Support executing Spark from syml...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1875#issuecomment-5203 QA results for PR 1875:- This patch FAILED unit tests.For more information see test ouptut:https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18953/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update global variables of HttpBroadcast so th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2058#issuecomment-52780307 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18954/consoleFull) for PR 2058 at commit [`a1b4efb`](https://github.com/apache/spark/commit/a1b4efb9a236a64d40c76f60ec707c7aa6bd5989). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class HttpBroadcastContainer extends Serializable with Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update global variables of HttpBroadcast so th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2058#issuecomment-52780297 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18954/consoleFull) for PR 2058 at commit [`a1b4efb`](https://github.com/apache/spark/commit/a1b4efb9a236a64d40c76f60ec707c7aa6bd5989). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update global variables of HttpBroadcast so th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2058#issuecomment-52781729 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18955/consoleFull) for PR 2058 at commit [`2646398`](https://github.com/apache/spark/commit/26463987b7647dc8f09b01fdd1c323962e22422b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update global variables of HttpBroadcast so th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2058#issuecomment-52781827 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18955/consoleFull) for PR 2058 at commit [`2646398`](https://github.com/apache/spark/commit/26463987b7647dc8f09b01fdd1c323962e22422b). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class HttpBroadcastContainer extends Serializable with Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3139] Akka timeouts from ContextCl...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2056#issuecomment-52783094 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18956/consoleFull) for PR 2056 at commit [`f388e6d`](https://github.com/apache/spark/commit/f388e6daef1762efad2c80c9546f1970cbce5e2c). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3139] Akka timeouts from ContextCl...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2056#issuecomment-52783109 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18956/consoleFull) for PR 2056 at commit [`f388e6d`](https://github.com/apache/spark/commit/f388e6daef1762efad2c80c9546f1970cbce5e2c). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3148] Update global variables of HttpBr...
GitHub user YanTangZhai opened a pull request: https://github.com/apache/spark/pull/2059 [SPARK-3148] Update global variables of HttpBroadcast so that multiple SparkContexts can coexist Update global variables of HttpBroadcast so that multiple SparkContexts can coexist You can merge this pull request into a Git repository by running: $ git pull https://github.com/YanTangZhai/spark SPARK-3148 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2059.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2059 commit cdef539abc5d2d42d4661373939bdd52ca8ee8e6 Author: YanTangZhai Date: 2014-08-06T13:07:08Z Merge pull request #1 from apache/master update commit cbcba66ad77b96720e58f9d893e87ae5f13b2a95 Author: YanTangZhai Date: 2014-08-20T13:14:08Z Merge pull request #3 from apache/master Update commit e751ebd22c0683746b8e13c48570bf22b4de45db Author: yantangzhai Date: 2014-08-20T14:07:57Z [SPARK-3148] Update global variables of HttpBroadcast so that multiple SparkContexts can coexist commit 97b34079b4af178ff2bca42c314aeb0e51687167 Author: yantangzhai Date: 2014-08-20T14:11:34Z [SPARK-3148] Update global variables of HttpBroadcast so that multiple SparkContexts can coexist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update global variables of HttpBroadcast so th...
Github user YanTangZhai commented on the pull request: https://github.com/apache/spark/pull/2058#issuecomment-52783462 #2059 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update global variables of HttpBroadcast so th...
Github user YanTangZhai closed the pull request at: https://github.com/apache/spark/pull/2058 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
