[GitHub] spark pull request: [SPARK-3694] RDD and Task serialization debugg...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/3518#discussion_r21786759
  
--- Diff: 
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -458,6 +458,17 @@ private[spark] class TaskSetManager(
   val startTime = clock.getTime()
   // We rely on the DAGScheduler to catch non-serializable 
closures and RDDs, so in here
   // we assume the task can be serialized without exceptions.
+
+  // Check if serialization debugging is enabled
+  val debugSerialization: Boolean = sched.sc.getConf.
+getBoolean("spark.serializer.debug", false)
+  
+  // If enabled, print out the added JARs and files (as part of 
the context)
--- End diff --

I do not believe that added JARs / files should have any bearing on whether 
something serializes correctly (although they could impact _deserialization_).  
Therefore, I don't think it makes sense to print this trace information via 
this debug option.

Also: the user can easily trace this themselves by just logging 
`sc.addedFiles` or `addedJars`, so I don't think that we need to have per-task 
/ RDD logging of this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3694] RDD and Task serialization debugg...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/3518#discussion_r21786754
  
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 
---
@@ -873,6 +943,10 @@ class DAGScheduler(
   // We've already serialized RDDs and closures in taskBinary, but 
here we check for all other
   // objects such as Partition.
   try {
+if(debugSerialization)
--- End diff --

Please see our code style guide; there should be spaces after the `if` and 
the brace should be included on this line.

https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3694] RDD and Task serialization debugg...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/3518#discussion_r21786752
  
--- Diff: 
core/src/main/scala/org/apache/spark/util/SerializationHelper.scala ---
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import java.io.NotSerializableException
+import java.nio.ByteBuffer
+
+import scala.collection.mutable.HashMap
+import scala.util.control.NonFatal
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.scheduler.Task
+import org.apache.spark.serializer.SerializerInstance
+
+object SerializationState extends Enumeration {
+  // Define vars to standardize debugging output
+  type SerializationState = String
+  var Failed = "Failed to serialize parent."
--- End diff --

Why are these vars?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3694] RDD and Task serialization debugg...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/3518#discussion_r21786744
  
--- Diff: docs/configuration.md ---
@@ -517,6 +517,14 @@ Apart from these, the following properties are also 
available, and may be useful
   
 
 
+  spark.serializer.debug
+  false
+  
+To view the dependency graph for an RDD or the file dependencies for 
task set this option
+to true. Doing so will display a text based graph of these 
dependencies along with a
+serialization trace that identifies which components of an RDD failed 
to serialize.  
--- End diff --

Is this true?  The original error message tells me the class of the object 
that failed to serialize.  This comment suggests that this trace tells me 
something like which _instance_ that caused the error, but unless I'm mistaken 
I don't think this patch actually does that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3694] RDD and Task serialization debugg...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/3518#issuecomment-66868725
  
IMO, this patch still needs a lot of work before it will be ready to merge. 
 I'm not convinced that telling me which RDD referenced the unserializable 
object, by itself, will be a helpful debugging tool.  In many cases, it's 
obvious which object is non-serializable: for instance, say that I try to 
serialize a database connection pool instance.  If it's an 
explicitly-referenced user-created object, then it's usually not too hard to 
find out the source of the reference.  The hard cases are where implicit 
references to non-serializable objects like SparkContext have been included in 
the closure.  In these cases, I might only have one RDD in my dependency chain 
and still run into serialization issues, in which case I don't feel that this 
patch's current approach will be very helpful to me for debugging.  It would be 
much more useful to print a chain of references to a non-serializable object of 
the appropriate type.

Do you mind running some examples in the `spark-shell` and pasting the 
output generated by this patch?  This would help me and other reviewers to 
asses whether this patch's current approach is useful.

There are also many code style issues here, but I don't want to spend too 
much time commenting on them before we make sure that the high-level approach 
is okay.

Other reviewers: please take a look at the JIRA and chime in here.  Do you 
think that this patch's current functionality is useful, or should we block / 
wait in favor of a more full-featured solution?  I think that we have plenty of 
time before 1.3.0, so I'm in favor of taking more time to implement a more 
full-featured approach since I don't think we're in a huge rush for this 
feature.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4839] Adding a documentation about dyna...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3689#issuecomment-66868521
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4839] Adding a documentation about dyna...

[oza]

GitHub user oza opened a pull request:

https://github.com/apache/spark/pull/3689

[SPARK-4839] Adding a documentation about dynamic resource allocation.

Signed-off-by: Tsuyoshi Ozawa 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/oza/spark SPARK-4839

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/3689.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3689


commit abc7e22fb95db2abd610a32dc3cb3dd3745fadb6
Author: Tsuyoshi Ozawa 
Date:   2014-12-13T07:34:25Z

Adding a documentation about dynamic resource allocation.

Signed-off-by: Tsuyoshi Ozawa 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4006] Block Manager - Double Register C...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2854#issuecomment-66868340
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24433/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4006] Block Manager - Double Register C...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2854#issuecomment-66868337
  
  [Test build #24433 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24433/consoleFull)
 for   PR 2854 at commit 
[`95ae4db`](https://github.com/apache/spark/commit/95ae4dbe090f16e863614a2f4f2e0b0fb90ec1af).
 * This patch **fails some tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3967] don’t redundantly overwrite exe...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/2848#issuecomment-66868021
  
Hmm, maybe a legitimate test failure?

```
[info] - Distributing files locally security On *** FAILED *** (82 
milliseconds)
[info]   java.io.FileNotFoundException: 
/tmp/spark-27c67e3e-d5f5-4d86-b9fa-948319ea6b7a/test/FileServerSuite.txt (No 
such file or directory)
[info]   at java.io.FileInputStream.open(Native Method)
[info]   at java.io.FileInputStream.(FileInputStream.java:146)
[info]   at 
com.google.common.io.Files$FileByteSource.openStream(Files.java:127)
[info]   at 
com.google.common.io.Files$FileByteSource.openStream(Files.java:117)
[info]   at com.google.common.io.ByteSource.copyTo(ByteSource.java:247)
[info]   at com.google.common.io.Files.copy(Files.java:458)
[info]   at 
org.apache.spark.HttpFileServer.addFileToDir(HttpFileServer.scala:72)
[info]   at org.apache.spark.HttpFileServer.addFile(HttpFileServer.scala:55)
[info]   at org.apache.spark.SparkContext.addFile(SparkContext.scala:965)
[info]   at 
org.apache.spark.FileServerSuite$$anonfun$3.apply$mcV$sp(FileServerSuite.scala:96)
[info]   at 
org.apache.spark.FileServerSuite$$anonfun$3.apply(FileServerSuite.scala:90)
[info]   at 
org.apache.spark.FileServerSuite$$anonfun$3.apply(FileServerSuite.scala:90)
[info]   at 
org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
[info]   at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
[info]   at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
[info]   at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555)
[info]   at 
org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
[info]   at 
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
[info]   at 
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[...]
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4006] Block Manager - Double Register C...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2854#issuecomment-66867662
  
  [Test build #24433 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24433/consoleFull)
 for   PR 2854 at commit 
[`95ae4db`](https://github.com/apache/spark/commit/95ae4dbe090f16e863614a2f4f2e0b0fb90ec1af).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4006] Block Manager - Double Register C...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/2854#issuecomment-66867614
  
another flaky test. retest this please 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66866916
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24432/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66866915
  
  [Test build #24432 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24432/consoleFull)
 for   PR 3564 at commit 
[`79d3c38`](https://github.com/apache/spark/commit/79d3c38fc7730d4c7f9753e7631d7d100df18c8f).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-1507][YARN]specify num of cores for AM

[XuTingjun]

Github user XuTingjun commented on the pull request:

https://github.com/apache/spark/pull/3686#issuecomment-66866834
  
I have tested, it works in yarn-client and yarn-cluster mode.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66866635
  
  [Test build #24432 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24432/consoleFull)
 for   PR 3564 at commit 
[`79d3c38`](https://github.com/apache/spark/commit/79d3c38fc7730d4c7f9753e7631d7d100df18c8f).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[nchammas]

Github user nchammas commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66866615
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-4297 [BUILD] Build warning fixes omnibus

[nchammas]

Github user nchammas commented on the pull request:

https://github.com/apache/spark/pull/3157#issuecomment-66866588
  
+1 on this kind of cleanup work.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3967] don’t redundantly overwrite exe...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2848#issuecomment-66866422
  
  [Test build #24431 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24431/consoleFull)
 for   PR 2848 at commit 
[`bcf2543`](https://github.com/apache/spark/commit/bcf2543de693ce6bf8a9294e0af028d2e1be3ccc).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3967] don’t redundantly overwrite exe...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2848#issuecomment-66866426
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24431/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP] Remove many uses of Thread.sleep() from ...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/3687#issuecomment-66865631
  
/cc @tdas.  This is a work-in-progress towards removing most of the 
Thread.sleep() calls.  I'm making slow-but-steady progress; would love your 
feedback + any suggestions for other types of test restructuring / cleanup to 
improve stability.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4837] NettyBlockTransferService should ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3688#issuecomment-66865449
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24430/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4837] NettyBlockTransferService should ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3688#issuecomment-66865446
  
  [Test build #24430 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24430/consoleFull)
 for   PR 3688 at commit 
[`ebd2007`](https://github.com/apache/spark/commit/ebd2007d82ea909b1d060114148e4abc4f749b25).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3967] don’t redundantly overwrite exe...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2848#issuecomment-66865189
  
  [Test build #24431 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24431/consoleFull)
 for   PR 2848 at commit 
[`bcf2543`](https://github.com/apache/spark/commit/bcf2543de693ce6bf8a9294e0af028d2e1be3ccc).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[nchammas]

Github user nchammas commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66865164
  
:+1: Thank you for getting to the bottom of these flaky tests.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3967] don’t redundantly overwrite exe...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/2848#issuecomment-66865120
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66865097
  
I've opened #3687 to try to systematically address the causes of streaming 
test flakiness.  In the process, I uncovered what I think is a new bug in 
streaming checkpoint recovery.  Once I get that PR finished and merged, we 
should try re-testing this to see whether it fixes the failures that we've seen 
here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3607#issuecomment-66864596
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24429/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3607#issuecomment-66864594
  
  [Test build #24429 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24429/consoleFull)
 for   PR 3607 at commit 
[`7fa9e2e`](https://github.com/apache/spark/commit/7fa9e2e26f3de07ce7934cf623d57583fa917611).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class Analyzer(catalog: Catalog, registry: FunctionRegistry, 
caseSensitive: Boolean)`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4837] NettyBlockTransferService should ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3688#issuecomment-66864203
  
  [Test build #24430 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24430/consoleFull)
 for   PR 3688 at commit 
[`ebd2007`](https://github.com/apache/spark/commit/ebd2007d82ea909b1d060114148e4abc4f749b25).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4837] NettyBlockTransferService should ...

[aarondav]

GitHub user aarondav opened a pull request:

https://github.com/apache/spark/pull/3688

[SPARK-4837] NettyBlockTransferService should use spark.blockManager.port 
config

This is used in NioBlockTransferService here:

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/network/nio/NioBlockTransferService.scala#L66

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aarondav/spark SPARK-4837

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/3688.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3688


commit ebd2007d82ea909b1d060114148e4abc4f749b25
Author: Aaron Davidson 
Date:   2014-12-13T03:57:23Z

[SPARK-4837] NettyBlockTransferService should use spark.blockManager.port 
config




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4006] Block Manager - Double Register C...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2854#issuecomment-66863703
  
  [Test build #24428 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24428/consoleFull)
 for   PR 2854 at commit 
[`95ae4db`](https://github.com/apache/spark/commit/95ae4dbe090f16e863614a2f4f2e0b0fb90ec1af).
 * This patch **fails some tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4006] Block Manager - Double Register C...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2854#issuecomment-66863706
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24428/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...

[suyanNone]

Github user suyanNone commented on the pull request:

https://github.com/apache/spark/pull/3541#issuecomment-66863621
  
To be frankly, I am fresh in spark.
If have local fileSystem accessed errors, should we identify and mark as 
host-failed?  may wrapAs ReadFileException or WriteFileException or sth
and I not sure is work for distinguish from exception in spark...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3409#issuecomment-66863203
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24427/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3409#issuecomment-66863200
  
  [Test build #24427 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24427/consoleFull)
 for   PR 3409 at commit 
[`08f44a7`](https://github.com/apache/spark/commit/08f44a7a6c8086071fc9e136bf3ffd2b02a2dd27).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class Analyzer(catalog: Catalog, registry: FunctionRegistry, 
caseSensitive: Boolean)`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3607#issuecomment-66863118
  
  [Test build #24429 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24429/consoleFull)
 for   PR 3607 at commit 
[`7fa9e2e`](https://github.com/apache/spark/commit/7fa9e2e26f3de07ce7934cf623d57583fa917611).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

[WangTaoTheTonic]

Github user WangTaoTheTonic commented on the pull request:

https://github.com/apache/spark/pull/3607#issuecomment-66863070
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4006] Block Manager - Double Register C...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2854#issuecomment-66862989
  
  [Test build #24428 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24428/consoleFull)
 for   PR 2854 at commit 
[`95ae4db`](https://github.com/apache/spark/commit/95ae4dbe090f16e863614a2f4f2e0b0fb90ec1af).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4006] Block Manager - Double Register C...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/2854#issuecomment-66862967
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [PySpark] Fix tests with Python 2.6 in 1.0 bra...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/3668#issuecomment-66862913
  
Nice! I'm merging this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4527][SQl]Add BroadcastNestedLoopJoin o...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3395#issuecomment-66862821
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24425/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4527][SQl]Add BroadcastNestedLoopJoin o...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3395#issuecomment-66862819
  
  [Test build #24425 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24425/consoleFull)
 for   PR 3395 at commit 
[`ea0e495`](https://github.com/apache/spark/commit/ea0e495bd35af287efbad53f7549311a25f916ad).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3607#issuecomment-66862423
  
  [Test build #24426 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24426/consoleFull)
 for   PR 3607 at commit 
[`7fa9e2e`](https://github.com/apache/spark/commit/7fa9e2e26f3de07ce7934cf623d57583fa917611).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3607#issuecomment-66862427
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24426/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Remove many uses of Thread.sleep() from stream...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3687#issuecomment-66862227
  
  [Test build #24424 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24424/consoleFull)
 for   PR 3687 at commit 
[`ad0056b`](https://github.com/apache/spark/commit/ad0056bf912d1d29cc2aa81be4615f527466bb1a).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  class StreamingTestWaiter(ssc: StreamingContext) `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Remove many uses of Thread.sleep() from stream...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3687#issuecomment-66862230
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24424/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3222#issuecomment-66861467
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24423/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3222#issuecomment-66861462
  
  [Test build #24423 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24423/consoleFull)
 for   PR 3222 at commit 
[`3980c8a`](https://github.com/apache/spark/commit/3980c8a7ca9e13d4b7c65d6dfe895518ea15d6ec).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class DBN(val stackedRBM: StackedRBM, val nn: MLP)`
  * `class MLP(`
  * `class RBM(`
  * `class StackedRBM(val innerRBMs: Array[RBM])`
  * `case class MinstItem(label: Int, data: Array[Int]) `
  * `class MinstDatasetReader(labelsFile: String, imagesFile: String)`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3222#issuecomment-66861201
  
  [Test build #24422 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24422/consoleFull)
 for   PR 3222 at commit 
[`0e81958`](https://github.com/apache/spark/commit/0e81958130df43f4aac55f3d57f3c20ef17c62e8).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class DBN(val stackedRBM: StackedRBM, val nn: MLP)`
  * `class MLP(`
  * `class RBM(`
  * `class StackedRBM(val innerRBMs: Array[RBM])`
  * `case class MinstItem(label: Int, data: Array[Int]) `
  * `class MinstDatasetReader(labelsFile: String, imagesFile: String)`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3222#issuecomment-66861205
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24422/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3409#issuecomment-66860916
  
  [Test build #24427 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24427/consoleFull)
 for   PR 3409 at commit 
[`08f44a7`](https://github.com/apache/spark/commit/08f44a7a6c8086071fc9e136bf3ffd2b02a2dd27).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

[zhzhan]

Github user zhzhan commented on the pull request:

https://github.com/apache/spark/pull/3409#issuecomment-66860869
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Remove many uses of Thread.sleep() from stream...

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/3687#issuecomment-66860487
  
I expect this to fail due to 
https://issues.apache.org/jira/browse/SPARK-4835.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

[WangTaoTheTonic]

Github user WangTaoTheTonic commented on the pull request:

https://github.com/apache/spark/pull/3607#issuecomment-66860231
  
@andrewor14 As `ClientArguments` didn't implement Log class before, I use 
println to warn.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-1953][YARN]yarn client mode Application...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3607#issuecomment-66860143
  
  [Test build #24426 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24426/consoleFull)
 for   PR 3607 at commit 
[`7fa9e2e`](https://github.com/apache/spark/commit/7fa9e2e26f3de07ce7934cf623d57583fa917611).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4527][SQl]Add BroadcastNestedLoopJoin o...

[wangxiaojing]

Github user wangxiaojing commented on the pull request:

https://github.com/apache/spark/pull/3395#issuecomment-66860102
  
@liancheng Sorry for the delay in my styling issues, and  thanks for  your 
help.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4527][SQl]Add BroadcastNestedLoopJoin o...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3395#issuecomment-66859945
  
  [Test build #24425 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24425/consoleFull)
 for   PR 3395 at commit 
[`ea0e495`](https://github.com/apache/spark/commit/ea0e495bd35af287efbad53f7549311a25f916ad).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Remove many uses of Thread.sleep() from stream...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3687#issuecomment-66859936
  
  [Test build #24424 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24424/consoleFull)
 for   PR 3687 at commit 
[`ad0056b`](https://github.com/apache/spark/commit/ad0056bf912d1d29cc2aa81be4615f527466bb1a).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Remove many uses of Thread.sleep() from stream...

[JoshRosen]

GitHub user JoshRosen opened a pull request:

https://github.com/apache/spark/pull/3687

Remove many uses of Thread.sleep() from streaming tests

This is a work-in-progress PR towards removing many of the `Thread.sleep()` 
calls from Spark Streaming's test suite, since I think that these calls make 
these tests race-prone and flaky.

I expect these test runs to fail, actually, since I uncovered some new bugs 
in Streaming.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JoshRosen/spark streaming-test-flakiness

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/3687.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3687


commit 303e828225630d2d08e14093db030e1b22fd7a27
Author: Josh Rosen 
Date:   2014-12-12T23:15:59Z

Add StreamingTestWaiter class

commit f8f6c93675e7989d3a81cb358b6690b09a063f74
Author: Josh Rosen 
Date:   2014-12-12T23:17:36Z

Remove (all but one) sleep calls from “socket input stream” test

commit 6ce068164c1c2c59c6a02b287859128d67eb8457
Author: Josh Rosen 
Date:   2014-12-12T23:41:08Z

Refactor several tests to use verifyOutput()

commit 5c31b8acd0314cd908077db575a96d664edb8abf
Author: Josh Rosen 
Date:   2014-12-12T23:59:23Z

Revert "Remove (all but one) sleep calls from “socket input stream” 
test"

This reverts commit f8f6c93675e7989d3a81cb358b6690b09a063f74.

Conflicts:

streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala

commit ad0056bf912d1d29cc2aa81be4615f527466bb1a
Author: Josh Rosen 
Date:   2014-12-13T00:42:01Z

WIP towards improved CheckpointSuite sleep logic.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3409#issuecomment-66859399
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24421/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3409#issuecomment-66859390
  
  [Test build #24421 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24421/consoleFull)
 for   PR 3409 at commit 
[`08f44a7`](https://github.com/apache/spark/commit/08f44a7a6c8086071fc9e136bf3ffd2b02a2dd27).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class Analyzer(catalog: Catalog, registry: FunctionRegistry, 
caseSensitive: Boolean)`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3222#issuecomment-66858770
  
  [Test build #24423 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24423/consoleFull)
 for   PR 3222 at commit 
[`3980c8a`](https://github.com/apache/spark/commit/3980c8a7ca9e13d4b7c65d6dfe895518ea15d6ec).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [WIP][SPARK-4251][SPARK-2352][MLLIB]Add RBM, A...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3222#issuecomment-66858513
  
  [Test build #24422 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24422/consoleFull)
 for   PR 3222 at commit 
[`0e81958`](https://github.com/apache/spark/commit/0e81958130df43f4aac55f3d57f3c20ef17c62e8).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4618][SQL] Make foreign DDL commands op...

[scwf]

Github user scwf commented on the pull request:

https://github.com/apache/spark/pull/3470#issuecomment-66858359
  
@marmbrus, updated, is this ok


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3409#issuecomment-66855785
  
  [Test build #24421 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24421/consoleFull)
 for   PR 3409 at commit 
[`08f44a7`](https://github.com/apache/spark/commit/08f44a7a6c8086071fc9e136bf3ffd2b02a2dd27).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

[zhzhan]

Github user zhzhan commented on the pull request:

https://github.com/apache/spark/pull/3409#issuecomment-66855013
  
Warning is OK to me, but I am against throwing exception. Will update it 
with logWarning("spark.yarn.am.extraJavaOptions will not take effect in driver 
mode")


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3672#issuecomment-66854192
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24420/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3672#issuecomment-66854184
  
  [Test build #24420 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24420/consoleFull)
 for   PR 3672 at commit 
[`31a6520`](https://github.com/apache/spark/commit/31a6520aea59f12d1a6896db6079c5cea62659a3).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[nchammas]

Github user nchammas commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66853045
  
FYI, [I've posted a question on Stack 
Overflow](http://stackoverflow.com/questions/27453882/how-can-run-tests-in-parallel-but-get-neatly-ordered-test-output)
 about the interleaved output.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...

[judynash]

Github user judynash commented on the pull request:

https://github.com/apache/spark/pull/3672#issuecomment-66843978
  
Fixed the styling issue and generated SQL Programming Guide doc. See 
screenshot http://i.imgur.com/l9F4mXq.png 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3672#issuecomment-66843487
  
  [Test build #24420 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24420/consoleFull)
 for   PR 3672 at commit 
[`31a6520`](https://github.com/apache/spark/commit/31a6520aea59f12d1a6896db6079c5cea62659a3).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66842034
  
A couple of those are [known flaky 
tests](https://issues.apache.org/jira/issues/?jql=labels%20%3D%20flaky-test).  
Specifically, it looks like you've run into 
[SPARK-4826](https://issues.apache.org/jira/browse/SPARK-4826) for the failures 
in WriteAheadLogBackedBlockRDDSuite.

The `org.apache.spark.streaming.CheckpointSuite.recovery with 
saveAsHadoopFiles operation` test has now failed twice with the same error, so 
I'll look at that test to see if I can spot any problems.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[nchammas]

Github user nchammas commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66841320
  
Hmm, [lots of 
failures](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24419/testReport/)
 in this latest test, mostly related to `streaming`, and one related to 
`network.nio`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3405] add subnet-id and vpc-id options ...

[jontg]

Github user jontg commented on the pull request:

https://github.com/apache/spark/pull/2872#issuecomment-66840618
  
@tylerprete That might occur if your VPC is not set up to auto-assign DNS 
records.  If you can, that is where I would suggest beginning an investigation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66840421
  
  [Test build #24419 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24419/consoleFull)
 for   PR 3564 at commit 
[`79d3c38`](https://github.com/apache/spark/commit/79d3c38fc7730d4c7f9753e7631d7d100df18c8f).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66840435
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24419/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66838546
  
  [Test build #24419 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24419/consoleFull)
 for   PR 3564 at commit 
[`79d3c38`](https://github.com/apache/spark/commit/79d3c38fc7730d4c7f9753e7631d7d100df18c8f).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[shaneknapp]

Github user shaneknapp commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66838179
  
thanks for the patience @nchammas !  :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[nchammas]

Github user nchammas commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66838119
  
Okie doke @shaneknapp.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[nchammas]

Github user nchammas commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66838083
  
Thanks for sending the update to `SparkSubmitSuite` @JoshRosen.

I also made a change in 79d3c38fc7730d4c7f9753e7631d7d100df18c8f to how the 
forked JVMs buffer their output. It seems that sbt 0.13 supports output 
buffering out of the box for parallelized tests via the [`logBuffered in 
Test`](http://www.scala-sbt.org/release/docs/Testing.html) setting, so it may 
just be a matter of understanding why that setting isn't taking.

Regarding whether the streaming tests are just failing due to flakiness, we 
did have a [single successful 
run](https://github.com/apache/spark/pull/3564#issuecomment-66747409) of all 
tests early this morning, and in 33 minutes at that. But yeah let's see what 
happens after we retry.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[shaneknapp]

Github user shaneknapp commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66838097
  
jenkins, test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66838015
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24418/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[shaneknapp]

Github user shaneknapp commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66837976
  
i'm gonna kill the 22418 build and retrigger.  need to restart jenkins 
really quick.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...

[judynash]

Github user judynash commented on a diff in the pull request:

https://github.com/apache/spark/pull/3672#discussion_r21774497
  
--- Diff: docs/sql-programming-guide.md ---
@@ -938,6 +938,18 @@ Configuration of Hive is done by placing your 
`hive-site.xml` file in `conf/`.
 
 You may also use the beeline script that comes with Hive.
 
+Thrift JDBC server also support sending thrift RPC messages over HTTP 
transport. 
+Use the following setting to enable HTTP mode as system property or in 
`hive-site.xml` file in `conf/`: 
+
+hive.server2.transport.mode - Set this to `http` 
+hive.server2.thrift.http.port - HTTP port number fo listen on; default 
is 10001
--- End diff --

Rendered the site and fixed few issues caused by ">" chars. 
Final image can be found here: http://imgur.com/LPceQ9J



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66837328
  
  [Test build #24417 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24417/consoleFull)
 for   PR 3564 at commit 
[`cc46a61`](https://github.com/apache/spark/commit/cc46a6105724f7d9cb0f824a52213f68bdae1b7c).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66837335
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24417/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66836113
  
  [Test build #24418 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24418/consoleFull)
 for   PR 3564 at commit 
[`79d3c38`](https://github.com/apache/spark/commit/79d3c38fc7730d4c7f9753e7631d7d100df18c8f).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[nchammas]

Github user nchammas commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66835945
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [PySpark] Fix tests with Python 2.6 in 1.0 bra...

[shaneknapp]

Github user shaneknapp commented on the pull request:

https://github.com/apache/spark/pull/3668#issuecomment-66835922
  
woot!  :)

On Fri, Dec 12, 2014 at 12:47 PM, UCB AMPLab 
wrote:
>
> Test PASSed.
> Refer to this link for build results (access rights to CI server needed):
> https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24414/
> Test PASSed.
>
> —
> Reply to this email directly or view it on GitHub
> .
>


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66835509
  
  [Test build #24417 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24417/consoleFull)
 for   PR 3564 at commit 
[`cc46a61`](https://github.com/apache/spark/commit/cc46a6105724f7d9cb0f824a52213f68bdae1b7c).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/1222#issuecomment-66833819
  
Failures are in streaming tests, which are unrelated to this pr.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/1222#issuecomment-66833605
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24415/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/1222#issuecomment-66833592
  
  [Test build #24415 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24415/consoleFull)
 for   PR 1222 at commit 
[`59c561c`](https://github.com/apache/spark/commit/59c561c284a38e067355564a984880d1d581fb96).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [PySpark] Fix tests with Python 2.6 in 1.0 bra...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3668#issuecomment-66832642
  
  [Test build #24414 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24414/consoleFull)
 for   PR 3668 at commit 
[`b32583d`](https://github.com/apache/spark/commit/b32583dee4e9850d201a11d035bb8912c158c38e).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [PySpark] Fix tests with Python 2.6 in 1.0 bra...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3668#issuecomment-66832650
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24414/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Test parallelization WIP

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/3681#issuecomment-66830291
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24416/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Test parallelization WIP

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/3681#issuecomment-66830275
  
  [Test build #24416 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24416/consoleFull)
 for   PR 3681 at commit 
[`3513e84`](https://github.com/apache/spark/commit/3513e84be3fa02473a21a798095f62a193b2360b).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/3409#issuecomment-66828213
  
LGTM.  I can go either way on the warning. It might be useful just to print 
a single line warning stating its not going to be used.  1 log line on executor 
startup doesn't seem to bad if they do include both in the spark-defaults.conf


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-3431] [WIP] Parallelize test execution

[JoshRosen]

Github user JoshRosen commented on the pull request:

https://github.com/apache/spark/pull/3564#issuecomment-66827468
  
The other two test failures _might_ be due to streaming flakiness, but I 
don't want to rule out the possibility that they could be legitimate failures.  
We'll know when we re-run.  The failed streaming tests from this last run were:

- [org.apache.spark.streaming.CheckpointSuite.recovery with 
saveAsHadoopFiles 
operation](https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24404/testReport/junit/org.apache.spark.streaming/CheckpointSuite/recovery_with_saveAsHadoopFiles_operation/)
- [org.apache.spark.streaming.StreamingContextSuite.stop 
gracefully](https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24404/testReport/junit/org.apache.spark.streaming/StreamingContextSuite/stop_gracefully/)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org